Taoffi's blog

prisonniers du temps

Json object explorer

[json objects =>to property bags =>to objects]

Reducing dependency between clients and services is a major common question in software solutions.

One important area of client/server dependencies lies in the structure of objects involved in exchanged messages (requests / responses). For instance: a new property inserted to an object on service side, often crashes the other side (client) until the new property is introduced on the involved object.

I previously posted about loose coupling through property bags abstractions.

My first approach was based on creating a common convention between service and client which implies transforming involved objects into property bags whose values would be assigned as needed to business objects at each side on runtime. That still seems to be a 'best solution' in my point of view.

Another approach is to transform the received objects (at either side: server/client) into property bags before assigning their values to the related objects.

This second approach is better suited for situations where creating a common convention would be difficult to put in place.

While working on some projects based on soap-xml messages, I wrote a simple transformer: [xml => property bags => objects].The transformer then helped write an xml explorer (which actually views xml content as its property bag tree. You can read about this in a previous post).

Another project presented a new challenge in that area, as the service (JEE) was using Json format for its messages. In collaboration with the Java colleagues, we could implement the property bag approach which helped ease client / server versioning issues.

A visual tool, similar to xml explorer, was needed for developers to explore json messages' structures. And that was time for me to write a new json <==>-property bag parser.

The goal was to:

  • Transform json content to property bags
  • Display the transformed property bag tree

Using Newtonsoft's Json library – notably its Linq extensions – was essential.

Hereafter the global dependency diagram of the Json explorer app:

 

The main method in the transformation is ParseJsonString to which you provide the string to be parsed.

Its logic is rather simple:

  • A json string is the representation of either:
    • An object:
      • Read its properties (which may contain arrays… see below)
    • Or an array of:
      • Objects: read the array's objects
      • Arrays: read the array's arrays

 

Code snippets

The parse json string method

public static PropertyBag ParseJsonString(string jsonString)
{
    JObject jObj = null;
    PropertyBag bag = new PropertyBag("Json");
    ObjProperty bagRootNode;
    JArray jArray = null;
    string exceptionString = "";

    /// the json string is either:
    /// * a json object
    /// * a json array
    /// * or an invalid string

    // try to parse the string as a JsonObject
    try
    {
        jObj = JObject.Parse(jsonString);
    }
    catch (Exception ex)
    {
        jObj = null;
        exceptionString = ex.Message;
    }

    // try to parse the string as a JArray
    if(jObj == null)
    {
        try
        {
            jArray = JArray.Parse(jsonString);
        }
        catch (Exception ex2)
        {
            jArray = null;
            exceptionString = ex2.Message;
        }

        if(jArray == null)
        {
            bag.Add(new ObjProperty(_exceptionString, null, false) { ValueAsString = exceptionString });
            return bag;
        }
    }

    bagRootNode = new ObjProperty("JsonRoot", null, false);

    if(bagRootNode.Children == null)
        bagRootNode.Children = new PropertyBag();

    bag.Add(bagRootNode);

    if(jObj != null)
    {
        bagRootNode.SourceDataType = typeof(JObject);
        bagRootNode.Children = ParseJsonObject(bagRootNode, jObj);
    }
    else if(jArray != null)
    {
        bagRootNode.SourceDataType = typeof(JArray);
        bagRootNode.Children = ParseJsonArray(bagRootNode, jArray);
    }

    return bag;
}

 

Parse json array code snippet

 

private static PropertyBag ParseJsonArray(ObjProperty parentItem, JArray jArray)
{
    if(parentItem == null || jArray == null)
        return null;

    ObjProperty childItem;

    if(parentItem.Children == null)
        parentItem.Children = new PropertyBag();

    PropertyBag curBag        = parentItem.Children;

    foreach(var item in jArray.Children())
    {
        JObject jo     = item as JObject;
        JArray subArray = item as JArray;
        PropertyBag childBag;
        Type nodeType = subArray != null ? typeof(JArray) : typeof(JObject);

        childItem    = new ObjProperty("item", parentItem, false) { SourceDataType = nodeType };

        if (jo != null)
            childBag = ParseJsonObject(childItem, jo);
        else if(subArray != null)
            childBag    = ParseJsonArray(childItem, subArray);
        else
            continue;

        curBag.Add(childItem);
    }

    return curBag;
}

 

Json to Xml

As, now, we have the json content in property bags, we can almost directly get the xml equivalent (see screenshot below).

The used sample json string: for the following screenshot:

 

{
    "web-app": {
    "servlet": [
    {
        "servlet-name": "cofaxCDS",
        "servlet-class": "org.cofax.cds.CDSServlet",
        "init-param": {
            "configGlossary:installationAt": "Philadelphia, PA",
            "configGlossary:adminEmail": "ksm@pobox.com",
            "configGlossary:poweredBy": "Cofax",
            …
            …
            "maxUrlLength": 500
            }
    },
    {
        "servlet-name": "cofaxEmail",
        "servlet-class": "org.cofax.cds.EmailServlet",
        "init-param": {
            "mailHost": "mail1",
            "mailHostOverride": "mail2"
            }
    },
    {
        "servlet-name": "cofaxAdmin",
        "servlet-class": "org.cofax.cds.AdminServlet"
    },

    {
        "servlet-name": "fileServlet",
        "servlet-class": "org.cofax.cds.FileServlet"
    },
    {
        "servlet-name": "cofaxTools",
        "servlet-class": "org.cofax.cms.CofaxToolsServlet",
        "init-param": {
            "templatePath": "toolstemplates/",
            "log": 1,
            …
            …
            "adminGroupID": 4,
            "betaServer": true
            }
        }
    ],
    "servlet-mapping": {
        "cofaxCDS": "/",
        "cofaxEmail": "/cofaxutil/aemail/*",
        "cofaxAdmin": "/admin/*",
        "fileServlet": "/static/*",
        "cofaxTools": "/tools/*"
        },

    "taglib": {
        "taglib-uri": "cofax.tld",
        "taglib-location": "/WEB-INF/tlds/cofax.tld"
        }
    }
}



Screenshot

 

You may download the binaries here!

The source code is available here!

Yet another [rather simple] OpenXML package explorer!

As its name implies, OpenXML is XML-centric technology (i.e. which extensively uses XML to describe the meta-model and contents of each and every element of a document).

OpenXML documents themselves (MS Word, Excel, PowerPoint…) are compressed packages that include numerous xml files which describe the document's meta-models and contents.

To explore these items, you can simply rename the document file to .zip and then navigate through this zip archive to view the elements that may be of interest.

After proceeding this way, you of course must not forget to rename the file back to its original extension (.docx, .xlsx… or whatever). When working on an OpenXML project, you obviously have to do this so many times a day, which becomes a somehow daunting task!

Some very good tools exist to explore OpenXML documents' structures… but most of them expose a logical view of the document rather than an xml-centric view. Which is of course very helpful… but does not apply when you just need to see the physical structure and the xml content of meta-models and elements.

Moreover, most of these tools are somehow out dated (2007-2009…) and many simply crash on most of our 'modern era' OSs!

Add some other good reasons to write another explorer:

It is august, it is raining and I don't know what to do this week end (while listening to Maria Joao Pires playing Mozart Sonatas!)

 

The purpose and tools?

What I need is:

  • Open an OpenXML document as a zip archive (without having to rename it to .zip!)
  • Explore the content of the archive (folders / files)
  • Explore / search the xml tree of xml files that may be found there

 

Adding a reference to System.IO.Compression and System.IO.Compression.FileSystem, you can (relatively easily) present the archive tree structure like in the following image:

I created a view model layer to map my needs to the zip archive objects of the System.IO.Compression namespace:

iZipEntryVM: a view model for zip archive entry which offers some useful properties related to my purpose.

iZipEntryVmList: a collection of the above object which offers static methods to create the tree of a zip archive file as well as a 'selected item' property and selection change event.

iOfficeVM: (singleton) entry point ('main view model' in a way) that orchestrates the basic commands and selection change events.

 

XML property bags explorer, again, to the rescue!

I wrote in a previous post about using PropertyBags to explore xml nodes.

The xml explorer objects, methods and controls written during this research are now packaged into an independent library that can be referenced in the current project.

What I have to do is just wire the selection event of an item in the archive tree to the xml explorer objects.

 

That is, when I click an xml file on the zip archive tree, the selection change event is fired. If the selected item is an xml content, it is opened and its stream is sent to the xml explorer object to display the node tree.

 

In practice:

  • On the TreeView control's selection changed event, the SelectedItem of the view model collection (iZipEntryVmList) is set.
  • The collection fires its own Selection changed event.
  • iOfficeVM subscribes to that event and send the entry stream to the xml explorer:

 

_myArchiveList.SelectedItemChanged += _list_SelectedItemChanged

private async void _list_SelectedItemChanged(object sender, iZipEntryVM selectedItem)
{
    ZipArchiveEntry    entry    = selectedItem.ZipEntry;

     if(entry == null)
       return;
 
    if(selectedItem.IsXmlFile)
        await Task.Run(() => OpenXmlFile(selectedItem));
}


 

async void OpenXmlFile(iZipEntryVM entryVm)
{
    ZipArchiveEntry    zentry    = entryVm.ZipEntry;
    var        bagVm    = XmlExplorer.Instance.PropertyBagVM;

    if(bagVm != null)
    {
        Stream            stream;
        await Task.Run(() =>
        {
            stream        = zentry.Open();
            // this will build the xml nodes' property bag tree
            bagVm.SourceStream = stream;
        });
    }
}



 

Apart from presenting the xml nodes' tree, XML explorer offers some other useful features:

  • Search for content values and/or node names
  • View / copy the xml string of a selected node
  • Export (save) the current OpenXML item to a file.

 

And, not less important, I no more have to rename a file to do that…!

Some screenshots

Explorer General View

Sample search results

 

Binaries and source code

You may download the binaries here

You may also download the source code here.

Back end – front end: Services loose coupling

Most mobile solutions are built and operate in a distributed architecture where services play a prominent role for providing data and applying business logic.

Let us take an example of a hotel reservation mobile app:

  • When the user specifies the desired target city, a back end service would provide the list of available managed/operated hotels at this place.
  • Once he or she specifies the desired stay period, the service would filter the above list to provide those having free rooms for that period.
  • When the user selects a room and applies for the reservation, the service would then register that reservation, proceeds to payment… etc.

It is hardly imaginable for such an app to operate independently from a back end service.

 

The price of distributed apps

In the previous example, the mobile app presents and manipulates several objects: the place (city/location…), the hotel, the room… etc.

Information about each of these objects is obtained from the service. The structures (metadata) of objects transmitted by the service should obviously match those handled by the mobile app.

In a typical (SOAP) situation, using Visual Studio for instance, your mobile app project would reference the service. Through the service's WSDL file, Visual Studio would generate the code for objects (classes) and operations defined by that referenced service.

If one of service's objects structure changes, you have to relaunch the process to update your app's service reference so that your objects' metadata be in sync with those defined by the service. If you don't do this, your app will simply crash at the first service call involving any unmatched object!

After each single change, deploying the solution's components (service / client app) become also a big hurdle.

In a few solutions, that may be desirable (or required). In most, the hurdle surpasses the benefits. It is hardly sustainable in a large solution. Actually it is not even sustainable when you are the sole developer of both components!

Resorting to use REST instead of SOAP (you may combine both) does not solve the difficulty, as that only changes the serialization format.

 

Using one single object

Let us imagine a service whose operations return always the same object. That would be quite handy independently of that object's serialization format.

In several large solutions I came to explore during last years, aggregating services responses into an abstract object, say, ServiceResponse was a common design practice. A given service operation response may look like (wsdl):

<element name="Operation1Response">
    <complexType>
        <sequence>
            <element name="Operation1Output"    type="Operation1OutputObject"/>
            <element name="ServiceStatus"     type="ServiceStatus"/>
        </sequence>
    </complexType>
</element>

 

With this, you know that any service operation you may call will always return a structure containing your expected object along with a service status object.

That approach seems good. Still it is going half way in abstraction, because you still should know the exact structure (metadata) of the expected object. And hence doesn't solve the evolution and deployment hassle: if, for some reason, you change the service's object structure, all your consumers should sync this change before deployment. The deployment of a new service version requires the deployment of a new consumers' versions… big hassle = increased risks.

 

Back to property bags!

I previously talked about property bags (here and here…).

A short reminder:

  • Objects (as we know till nowJ) are structures composed of: properties, methods and events.
  • An object can be considered as a set of properties (a property bag)
  • An object property can be defined as:
    • A name
    • A data type
    • A Value
  • Property value can be either:
    • A primitive type (value type or whatever we may consider as 'primitive'… string for instance)
    • A composite type: an object itself containing a set of properties (=property bag)

 

 

With that simple structure we are able to represent virtually any object in a strongly-typed manner.

 

Using property bags, we can imagine a unique structure for our service response. Which may look like this (wsdl):

<element name="ServiceResponse">
    <complexType>
        <sequence>
            <element name="OperationOutput"    type="PropertyBag"/>
            <element name="ServiceStatus"     type="PropertyBag"/>
        </sequence>
    </complexType>
</element>

 

Now, when we call any service operation, we know in advance that it will return an output property bag, accompanied with a status property bag.

Each element of our property bag being strongly typed (specifies its data type), we can easily convert the received bag to the expected object.

That conversion itself can be subject to a set of business rules. But that would be implemented 'once' independently of service or consumer versions and deployment constraints.

 

This approach doesn't eliminate the need for a unified business model. It enforces its usage.

By creating a loose coupling mechanism between services and consumers it allows more separation of concerns and minimizes evolution and deployment risks.

Exercise: service side code sample

Let us define service response as:

[DataContract(Namespace = "")]
public partial class ServicePropertyBagStatus : ServiceMessage

{
    PropertyBagContainer    _responseOutpput;

    [DataMember]

    public PropertyBagContainer ResponseOutput

    {

        get { return _responseOutpput; }

        set { _responseOutpput = value; }

    }
    …
    …

Our response structure would then look like (wsdl):

<complexType name="ServicePropertyBagStatus">
    <complexContent mixed="false">
        <extension base="ServiceMessage">
            <sequence>
                <element name="ResponseOutput" type="PropertyBagContainer"/>
            </sequence>
        </extension>
    </complexContent>
</complexType>

 

Sample code of a service operation (for reading a customer data) may look like this:

[OperationContract]
public ServicePropertyBagStatus GetCustomer(int customerId)

{
    return SvcUtilities.LoadCustomer(customerId);

}

Our service utilities module method LoadCustomer:

public static ServicePropertyBagStatus LoadCustomer(int customeId)
{
    ServicePropertyBagStatus    status    = new ServicePropertyBagStatus();

    if(customerId == 0)
    {
        status.SetErrorMessage("No value provided customerId");
        return status;
    }

    // load the customer's info from database server

    Customer        customer    = Customer.LoadDbUser(customerId);

    // extract this object's data contract into a property bag container

    status.ResponseOutput = ExtractPropertyBagContainer(customer.GetType(),customer);

    // return the service response status

    return status;
}

Extract the object's property bag:

internal static PropertyBagContainer ExtractPropertyBagContainer(Type type, object obj)
{
    // get soap xml of the object's data contract

    DataContractSerializer serializer = new DataContractSerializer(type);
    MemoryStream stream = new MemoryStream();

    serializer.WriteObject(stream, obj);

    // parse the soap xml into a property bag container
    PropertyBagContainer container = PropertyBagContainer.ParseXml(stream, true, null);

    stream.Close();
    return container;

}

Some Property bag container helper methods:

public class PropertyBagContainer
{
    protected PropertyBag _bag;


    // parse an object into a property bag container
    public static PropertyBagContainer ParseObject(object obj, ObjProperty parent);

    // parse xml stream into a property bag container
    public static PropertyBagContainer ParseXml(Stream stream, bool parseSiblings, ObjProperty parent);

    // parse xml node tree into a property bag container
    public static PropertyBagContainer ParseXml(XElement xnode, bool parseSiblings, ObjProperty parent);
}

Client side code sample

Our service operations will now return a Property Bag Container containing the expected object's property bag. A helper static method of PropertyBag (AssignObjectProperties) allows us to assign its content to a specific object:

public static bool AssignObjectProperties(object targetObject, PropertyBag bag)

 

We can thus write:

public bool ParsePropertyBag(PropertyBag bag)
{

return PropertyBag.AssignObjectProperties(this, bag);

}

 

Assigning property bag contents to an object is done by assigning values of the bag items to object's properties having the same name and data type. Extra logic can be introduced here according to your business requirements (for instance: checking object integrity through specific attributes).

 

Now, on the service consumer application, assume we have a service that is referenced as CustomerService. We may call the service's LoadCustomer operation like in the following code:

public bool LoadServiceCustomer(int customerId)
{
    CustomerService proxy = new CustomerService();
    var status = proxy.
LoadCustomer(customerId);


    // we may check the service status for errors here 
     ….

    // assign received properties to this object
    return this.ParsePropertyBag(status.ResponseOutput.PropertyBag);
}

Client side with Json

The process is the same in a REST configuration (You may see some details about WCF REST services here) as you actually will always receive a ServicePropertyBagStatus object independently of the transmission format (xml / json… etc.). Parsing the received response into a property bag container can be done using components like the .Net NewtonSoft.Json:

HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
string str = GetResponseString(request);
ServicePropertyBagStatus  response = JsonConvert.DeserializeObject<ServicePropertyBagStatus>(str);
PropertyBagContainer container    = response.ResponseOutput;

You can download the property bag binaries (std + portable) here.