Taoffi's blog

Objectives:

• Display a cloud of index words, each dimensioned relative to its occurrences in e-book information (e-book title, description, author, editor… etc.)
• On selection of a given word: display the list of e-book references related to the selected word
• On selection of an e-book: display its information details and all words linked to it

Context:

doc5ync web interface is based on a meta-model engine (simpleSite, currently being renamed to web5ync!).

I talked about meta-models in a past post Here, with some posts about its potential applications here.

The basic concept of meta-models is to describe an object by its set of properties and enable the user to act on these properties by modifying their values in the meta-model database. On runtime, those property values are assigned to each defined object.

In our case, for instance, we have a meta-model describing web page elements, and a meta-model describing the dataset of word index and their related e-books.

For web page elements, the approach considers a web page as a set of html tags (i.e. <div>, <table><tr><td>…, <img… etc.). Where each tag has a set of properties (style, and other attributes) for which you can define the desired values. On runtime, your meta-model-defined web page comes to life by loading its html tags, assigning to each the defined values and injecting the output of the process to the web response.

A dataset is similarly considered as a set of rows (obtained through a data source), each composed of data cells containing values. Data cells can then be either presented and manipulated through web elements (html tags, above) or otherwise manipulated through web services.

Data storage and relationships

As we mentioned in the previous post, index words and their related e-books are stored in database tables as illustrated in the following figure:

Each word of the index provides us with its number of occurrences in e-book text sequences (known on Trie scan).

Html formatting using a SQL view

To reflect this information into a presentation, we used a view to format a html div element for each word relative to its number of occurrences. The query looks like the following code

select
  w.id            as word_id
 , w.n_occurs
 , N'<div style="BASIC STYLE STRING HERE…; display:inline;'

-- add the font-size style relative to number of occurrences
+
case
    when w.n_occurs between 0 and 2    then N' font-size:10pt;'
    when w.n_occurs between 3 and 8    then N' font-size:14pt;'
    when w.n_occurs between 9 and 15    then N' font-size:16pt;'
    when w.n_occurs between 16 and 24    then N' font-size:22pt;'
    when w.n_occurs between 25 and 2147483647    then N' font-size:26pt; '
end

+ N'"'
-- add whatever html attributes we need (hover/click…)
+
N' id="div' + convert(nvarchar(32), w.id)
+ N'" onclick="select_data_cell(''' + convert(nvarchar(32), w.id) + N''');" '
     as word_string_html

-- add other columns if needed
from dbo.doc5_trie_words w
order by w.word_string

The above view code provides us with html-preformatted string for each word index in the data row.

Tweaking the data rows into a cloud of words

On a web page, a dataset is commonly displayed as a grid (table / columns / rows), and web5ync knows how to read a data source, and output its rows into that form. But that did not seem to be convenient in our case, because it simply displays index words each on a row which is not really the presentation we are looking for!

To resolve this, we simply need to change the dataset web container from a <table> (and its containing rows / cells) into <div> tags (with style=display: inline).

Here a sample of html code of the above presentation:

<table>
  <tr>
    <td>
    <div style="font-size:16pt;" onclick="select_data_cell('29008');">After</div>
  </td>
</tr>
<tr>
  <td>
    <div style="font-size:26pt; onclick="select_data_cell('28526');">after</div>
  </td>
</tr>

<!-- the table rows go on... --> 

And here is a sample html code of the presentation we are looking for:

<div style="display:inline;">
    <div id="td_word_string_html82" style="display:inline;">
       <div style="font-size:26pt;" onclick="select_data_cell('28526');">after</div>
      </div>
</div>

<div style="display:inline;">
    <div id="td_word_string_html83">
      <div style="font-size:16pt;" onclick="select_data_cell('29008');">After</div>
    </div>
</div>

<div style="display:inline;">
   <div style="display:inline;">
      <div style="font-size:10pt;" onclick="select_data_cell('17657');">AFTER</div>
    </div>
</div>

Which looks closer to what we want:

Interacting with index words

The second part of our task is to allow the user to interact with the index words: clicking a word = display its related e-books, clicking an e-book = display the e-book details + display index words specifically linked to that e-book.

For this, we are going to use a few of the convenient features of web5ync, namely: Master/details data binding, and Tabs. (I will write more about these features in a future post)

Web5ync master/details binding allows linking a subset of data to a selected item in the master section. Basically, each data section is an iframe. The event of selecting a data row in one iframe can update the document source of one or more iframes. All what we need is: 1. define a column that will be used as the row's id, and 2. define how the value of that id should be passed to the target iframe (typically: url parameter name).

Tabs are convenient in our case as they will allow distributing the information in several areas while optimizing web page space usage.

In the figure above, we have 3 main data tabs: n Explore by words, n Document info and n Selected document words.

On the first tab:

• clicking a word (in the upper iframe) should display the list of its related e-books (in lower iframe of that same tab)
• clicking an e-book row on the lower iframe should: first displays its details (an iframe on the second tab), and display all words directly linked to the selected e-book (an iframe in the 3^rd tab). (figures below)

In that last tab, we can play once again with the displayed words, to show other documents sharing one of them:

Most mobile solutions are built and operate in a distributed architecture where services play a prominent role for providing data and applying business logic.

Let us take an example of a hotel reservation mobile app:

• When the user specifies the desired target city, a back end service would provide the list of available managed/operated hotels at this place.
• Once he or she specifies the desired stay period, the service would filter the above list to provide those having free rooms for that period.
• When the user selects a room and applies for the reservation, the service would then register that reservation, proceeds to payment… etc.

It is hardly imaginable for such an app to operate independently from a back end service.

The price of distributed apps

In the previous example, the mobile app presents and manipulates several objects: the place (city/location…), the hotel, the room… etc.

Information about each of these objects is obtained from the service. The structures (metadata) of objects transmitted by the service should obviously match those handled by the mobile app.

In a typical (SOAP) situation, using Visual Studio for instance, your mobile app project would reference the service. Through the service's WSDL file, Visual Studio would generate the code for objects (classes) and operations defined by that referenced service.

If one of service's objects structure changes, you have to relaunch the process to update your app's service reference so that your objects' metadata be in sync with those defined by the service. If you don't do this, your app will simply crash at the first service call involving any unmatched object!

After each single change, deploying the solution's components (service / client app) become also a big hurdle.

In a few solutions, that may be desirable (or required). In most, the hurdle surpasses the benefits. It is hardly sustainable in a large solution. Actually it is not even sustainable when you are the sole developer of both components!

Resorting to use REST instead of SOAP (you may combine both) does not solve the difficulty, as that only changes the serialization format.

Using one single object

Let us imagine a service whose operations return always the same object. That would be quite handy independently of that object's serialization format.

In several large solutions I came to explore during last years, aggregating services responses into an abstract object, say, ServiceResponse was a common design practice. A given service operation response may look like (wsdl):

<element name="Operation1Response">
    <complexType>
        <sequence>
            <element name="Operation1Output"    type="Operation1OutputObject"/>
            <element name="ServiceStatus"     type="ServiceStatus"/>
        </sequence>
    </complexType>
</element>

With this, you know that any service operation you may call will always return a structure containing your expected object along with a service status object.

That approach seems good. Still it is going half way in abstraction, because you still should know the exact structure (metadata) of the expected object. And hence doesn't solve the evolution and deployment hassle: if, for some reason, you change the service's object structure, all your consumers should sync this change before deployment. The deployment of a new service version requires the deployment of a new consumers' versions… big hassle = increased risks.

Back to property bags!

I previously talked about property bags (here and here…).

A short reminder:

• Objects (as we know till nowJ) are structures composed of: properties, methods and events.
• An object can be considered as a set of properties (a property bag)
• An object property can be defined as:
  • A name
  • A data type
  • A Value
• Property value can be either:
  • A primitive type (value type or whatever we may consider as 'primitive'… string for instance)
  • A composite type: an object itself containing a set of properties (=property bag)

With that simple structure we are able to represent virtually any object in a strongly-typed manner.

Using property bags, we can imagine a unique structure for our service response. Which may look like this (wsdl):

<element name="ServiceResponse">
    <complexType>
        <sequence>
            <element name="OperationOutput"    type="PropertyBag"/>
            <element name="ServiceStatus"     type="PropertyBag"/>
        </sequence>
    </complexType>
</element>

Now, when we call any service operation, we know in advance that it will return an output property bag, accompanied with a status property bag.

Each element of our property bag being strongly typed (specifies its data type), we can easily convert the received bag to the expected object.

That conversion itself can be subject to a set of business rules. But that would be implemented 'once' independently of service or consumer versions and deployment constraints.

This approach doesn't eliminate the need for a unified business model. It enforces its usage.

By creating a loose coupling mechanism between services and consumers it allows more separation of concerns and minimizes evolution and deployment risks.

Exercise: service side code sample

Let us define service response as:

[DataContract(Namespace = "")]
public partial class ServicePropertyBagStatus : ServiceMessage
{
    PropertyBagContainer    _responseOutpput;

    [DataMember]

    public PropertyBagContainer ResponseOutput

    {

        get { return _responseOutpput; }

        set { _responseOutpput = value; }

    }
    …
    …

Our response structure would then look like (wsdl):

<complexType name="ServicePropertyBagStatus">
    <complexContent mixed="false">
        <extension base="ServiceMessage">
            <sequence>
                <element name="ResponseOutput" type="PropertyBagContainer"/>
            </sequence>
        </extension>
    </complexContent>
</complexType>

Sample code of a service operation (for reading a customer data) may look like this:

[OperationContract]
public ServicePropertyBagStatus GetCustomer(int customerId)
{
    return SvcUtilities.LoadCustomer(customerId);
}

Our service utilities module method LoadCustomer:

public static ServicePropertyBagStatus LoadCustomer(int customeId)
{
    ServicePropertyBagStatus    status    = new ServicePropertyBagStatus();

    if(customerId == 0)
    {
        status.SetErrorMessage("No value provided customerId");
        return status;
    }

    // load the customer's info from database server
    Customer        customer    = Customer.LoadDbUser(customerId);

    // extract this object's data contract into a property bag container
    status.ResponseOutput = ExtractPropertyBagContainer(customer.GetType(),customer);

    // return the service response status
    return status;
}

Extract the object's property bag:

internal static PropertyBagContainer ExtractPropertyBagContainer(Type type, object obj)
{
    // get soap xml of the object's data contract
    DataContractSerializer serializer = new DataContractSerializer(type);
    MemoryStream stream = new MemoryStream();

    serializer.WriteObject(stream, obj);

    // parse the soap xml into a property bag container
    PropertyBagContainer container = PropertyBagContainer.ParseXml(stream, true, null);

    stream.Close();
    return container;
}

Some Property bag container helper methods:

public class PropertyBagContainer
{
    protected PropertyBag _bag;

    // parse an object into a property bag container
    public static PropertyBagContainer ParseObject(object obj, ObjProperty parent);

    // parse xml stream into a property bag container
    public static PropertyBagContainer ParseXml(Stream stream, bool parseSiblings, ObjProperty parent);

    // parse xml node tree into a property bag container
    public static PropertyBagContainer ParseXml(XElement xnode, bool parseSiblings, ObjProperty parent);
}

Client side code sample

Our service operations will now return a Property Bag Container containing the expected object's property bag. A helper static method of PropertyBag (AssignObjectProperties) allows us to assign its content to a specific object:

public static bool AssignObjectProperties(object targetObject, PropertyBag bag)

We can thus write:

public bool ParsePropertyBag(PropertyBag bag)
{
return PropertyBag.AssignObjectProperties(this, bag);
}

Assigning property bag contents to an object is done by assigning values of the bag items to object's properties having the same name and data type. Extra logic can be introduced here according to your business requirements (for instance: checking object integrity through specific attributes).

Now, on the service consumer application, assume we have a service that is referenced as CustomerService. We may call the service's LoadCustomer operation like in the following code:

public bool LoadServiceCustomer(int customerId)
{
    CustomerService proxy = new CustomerService();
    var status = proxy. LoadCustomer(customerId);

    // we may check the service status for errors here 
     ….

    // assign received properties to this object
    return this.ParsePropertyBag(status.ResponseOutput.PropertyBag);
}

Client side with Json

The process is the same in a REST configuration (You may see some details about WCF REST services here) as you actually will always receive a ServicePropertyBagStatus object independently of the transmission format (xml / json… etc.). Parsing the received response into a property bag container can be done using components like the .Net NewtonSoft.Json:

HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
string str = GetResponseString(request);
ServicePropertyBagStatus  response = JsonConvert.DeserializeObject<ServicePropertyBagStatus>(str);
PropertyBagContainer container    = response.ResponseOutput;
…
…

You can download the property bag binaries (std + portable) here.

Intro

I joined an interesting presentation today about unit tests. Part of the presentation was related to dependency injection usage for unit testing. That brought to my mind again the benefits of meta-models.

The problem

Unit testing often faces the problem of having to instantiate objects that may impact the test itself or simply make the test impossible.

A sound example of this is when the method to be tested requires the instantiation of an object requiring, for instance, a database connection.

In this schema, we have 3 actors

• The function
• The caller
• The object (SomeObject) which cannot be instantiated.

Traditional recipe

One, now traditional, recipe is to use an Interface (instead of the specific object) as a function parameter. And, according to the context, instantiate and use an object that implements the defined interface. The figure below illustrates this.

Good solution… a little acrobatic, but OK… good.

A problem, though. We now have 6 actors (to code and maintain):

• The Function
• The caller
• The original 'true' object (SomeObject)
• The Interface
• The interface implementations (=2)!

On the other hand, testing a function may also need to instantiate a 'true' object context. In which case, we would have to rewrite (and recompileJ) our test code to instantiate a 'true' implementation when required.

A third point: in this solution, we defined the Interface to represent an abstraction of the original object. What if the original object itself was, for instance, a database record? This would then introduce a new actor in our playgroundJ

As Bjarne Stroustrup puts it:

"Any verbose and tedious solution is error-prone because programmers get bored" J

Meta-models may be a better way

In the current case, meta-models (see my previous posts) can be used to more simply describe all actors (object structures / methods / properties…).

An illustration:

Reminder: meta-models are closely related and enforced by Reflection (assemblies' meta-data). Through meta-models AND Reflection (namespace System.Reflection) developers can gain more flexibility to create versatile and scalable software.

In our present case, meta-models not only expose less actors to code and maintain, but also lower the complexity level of involved actors (for instance: maintaining database records instead of hard coding implementation).

The main goal for this approach is to ultimately write fewer universal methods to invoke and instantiate any object!

Will try to deliver a code sample in a future post. In the meantime, you may download meta-model sample code here!

In a previous post, I started exploring the TFS configuration database. Let's continue here by having a look at some more configuration objects and relationships.

TFS Config security objects

The relationship diagram above suggests that we can query user logins, group-membership for specific projects by a query like the following:

The above query may give us results similar to the following

I spent a long time during the last months writing about meta-models. That was in the context of a book for Editions ENI (France) about some concepts to which I dedicated some work since quite a while (it was around 1998, as long as I remember, that I started studying the basis of the approach).

In previous posts (see, for instance, Silverlight database to DataGrid, yet another approach- Part II) I exposed a generic approach for describing database meta-data. The direct practical purpose of those posts was reading and displaying data into Silverlight DataGrid. Through this exercise, we also saw how can our structures help us in integrating business rules at several level (meta-column, meta-table… and so on).

In this same exercise, applying the meta-models approach would consist of persisting these structures into the database… to read and create them dynamically.

We would, for instance, store our meta-columns / meta-tables definitions into the database and let our software objects load these structures and transform them into ‘live’ data, integrating all defined business and technical constraints.

What are meta-models?

I use the term "meta-model" to designate "a description of a description" stored in a database and interpreted (read and presented) by one or more (specific) software objects (classes).

That is: the information stored in the database describes an object or a behavior, and the software class that reads this information is aware of the fundamental semantics of this information and is, thus, able to interpret it independently of a particular context.

This means that, according to our view, a meta-model is composed of:

§  Database objects and structures to store the model's data;

§  Software (kernel) objects that know the fundamental semantics of the database model.

The collaborative aspect between the 'description' stored in the database and the 'runtime object' (which reads and interprets that description) is essential for this approach to be applicable.

Why use meta-models?

Let us take a sample application where the software should display a page (or form) composed of a Master / Detail connected regions… i.e. a list of items in a Master grid, with a Detail form that displays the details of the item selected in the Master grid.

That 'Master/Detail’ layout may be the software application's approach for presenting the information of several entities. The application developer may choose to create a specific page for each manipulated entity… a choice which may be correct in some contexts, but which also remains questionable by its repetitive approach.

In another, more automated, approach, the software developer may prepare templates for each entity's specific Master/Detail information, and load the required template according to the context's entity.

This latest approach would be a first step for a meta-modeling approach for entities' presentation.

In fact, storing these templates as part of software objects would make them dependent of their related objects' compiled state. That is: the template cannot evolve without changing and recompiling related objects' code. Storing template's information in a database (meta-model) would circumvent this inconvenience.

In the same time, storing templates' information in a database would also require a (more or less important) change in software object's structure and behavior. Using a database meta-model approach (to describe Master/Detail templates in our case), requires the software object to act in a more abstract way. And this is what we called 'collaborative' approach (between database meta-model and runtime objects).

Finally, in the long run, our software object would constitute a 'kernel object' capable of displaying and handling any entity's Master/Detail information stored in a database meta-model.

What can we do with meta-models?

The sample exposed above can of course be extended to more elaborate models and interactions between descriptions stored in a database and their software objects' counterpart.

Let us again take another practical sample: an application that may need to normalize the appearance of asp.net data grid controls.

We know that a GridView control contains some appearance properties like:

§  Width;

§  HeaderStyle-CssClass;

§  AlternatingRowStyle-BackColor;

§  … etc.

We may choose to use the database to store values for each of these properties and let our software objects assign these values to each DataGrid (on the page's load event for instance).

Now, to handle the appearance of our GridView controls, we just need to assign new values for these properties in the database. A task which requires less effort, less time and fewer technical skills.

Meta-models and Reflection

Reflection is the process of introspection (self-examining) of the internal structures of software objects at runtime. .NET implements Reflection in some classes defined in the System.Reflection namespace.

Our previous exercise can be extended to more abstract and useful applications.

In fact, a software object (like GridView in the last sample) has properties to which we can dynamically assign values if we may simply know their name:

After locating the property inside the specified object's Type (class) we may assign it a specified value:

Our meta-model may make use of Reflection mechanisms to store property names and values in the database, to read and assign the specified values to the specified objects at runtime.

This approach may be of interest in many software applications. Its main advantages are, again:

§  More adaptable and versatile software;

§  Reducing development and maintenance efforts, time and cost!

Meta-models and method invocation

What we explained above about using System.Reflection to discover an object's property by name and assign a value to the retrieved property, is also applicable to methods:

On the other side, user interface’s command elements like Menus, buttons… are often associated with an event handler that represents the entry point for the action of the command.

To explain how a meta-model can be useful in this area, let us take an example of an update button.

An update button collects the user entries into the form fields, checks their integrity and submits them to the server for updating the specified table;

Having the meta-data and business rule constraints for the columns of the entity, this update operation can be handled through one same handler. Or be handled by a dynamically selected handler specific to the context.

So, we know that a data entry form would have an update button. And we may need to assign a specific method as the handler of the Click event for this button.

All these required information elements can be stored into a database meta-model. The meta-model software would then:

§  Read (or otherwise find) the name of the method assigned as handler of the button's action event;

§  Dynamically locate the method in the software (loading the assembly when required);

§  Invoke the method.

Again, this meta-model solution can be applied on different contexts (records submission, web service operations invocation… etc.) and/or on different objects or controls (buttons, menus… etc.).

Interoperability and Integration

The database meta-model approach may also be used in a way similar to what we already use in Interop assemblies (which create links between 'unmanaged / native' code end 'managed code').

A meta-model-aware assembly method may, for instance, prepare entries to methods that reside in non-meta-model-aware assemblies or methods.

Such methods can also be defined in different projects independent of the 'meta-model-kernel' assembly. And may, thus, allow a high level integration of existing applications (or data) into the meta-model solution.

Extensibility

As we said above, the meta-model as exposed in this paper, is a collaborative ensemble where the database descriptions are tightly connected to the meta-model kernel software.

Two more questions though:

§  What about the evolution of the meta-model's own kernel?

§  How to describe objects that we don't yet know about?

Here comes another important point which seems to represent a good basis for answering the above questions:

§  An object is represented by a set of properties;

§  A property is, essentially, a definition composed of

·         A Name

·         And an Attribute (the Attribute defines property's constraints: mainly 'Data Type');

§  A Value can be assigned to a property in the context of either a class (static property) or to an instance of this class. (Note: Value should conform to the attribute's constraints).

So we may consider the data of an object as a collection of (Property / Value)… i.e. a collection of ((Name/attribute)/Value).

This seems to be, more or less easily, presented in a meta-model, which may allow us to handle future 'unknown objects' or future unknown 'properties'.

Meta-models self-describing

On one hand, meta-models are data stored in a database… and on the other, they are extensible. Managing them can also be described by meta-models. This recursive aspect can be of interest to some applications to allow users to autonomously manage software semantic behaviors without the need of extensive technical knowledge.