Taoffi's blog

prisonniers du temps

Jet.oledb.4.0 and utf8 bom story

As you may know, a text file may specify its encoding by a ‘bom’ (byte order mark) using several bytes at its beginning. For utf8 encoding the signature bytes are: 0xef, 0xbb, 0xbf.

I came across an issue while manipulating csv files, for which I decided to use utf8 (thinking that was a good choice for a multi-cultural environment!). The process involved reading and writing back to the same file after some insertions and updates. All using microsoft.jet.oledb.4.0 provider (with a schema.ini specifying CharacterSet=65001 (65001 being utf8 code page)).

My csv files had a header row of which the first column was, ironically, named ‘Match ID’. A few manipulations revealed a somehow strange behavior. Although the debugger showed that the first column’s name is ‘Match ID’, I could no more access ‘Match ID’ column by its name. Using the watch window, I asked the debugger:
myColumn.ColumnName == "Match ID"… it replied false… weird!

Viewing the column name's CharArray in hexa offers a more significant information:

 column name issue

That is evidently endless! With the time going, as you manipulate your columns and rewrite back to the csv file, you end up by having your 'Match ID' column prefixed by bytes from the utf8 bom code as many times you rewrite the csv file. And if you are a nice guy who lets the users reorder the columns as they need, you may end up by having all your columns affected by that issue!

column name bytes

Changing files’ encoding to unicode (whose bom signature is 0xff 0xfe 0xff 0xfe) does not reproduce the issue. Which makes it clear that the source of annoyance is not utf8 but rather the jet.oledb.4 data provider with utf8. Still, identifying the source is half way of solving the issue :).

How to solve this?

Well, you may think of ‘sanitizing’ your column names at every load! Which, in my view, does not seem quite practical.
In my case I just switched to unicode (despite more bytes waste!) to preserve multi-cultural data requirements.

xsl witness!

Transforming xml content through xsl stylesheets is a useful and relatively common feature in the development process. I talked about this in the previous post about OneNote pages html preview.
Searching in my personal code toolbox, I just found this ‘iXslWitness’, a tool I wrote a couple of years ago to check the effectiveness of a stylesheet in transforming xml to html. Its usage is quite simple: you select an xml file and the xsl stylesheet to use. And you get the html transformed content.
I hope that can be useful for anyone involved in such tasks!
A screenshot of transforming a OneNote page xml content (a list of ‘The World If’ publications of The Economist newspaper):

worldif2017-witness

You can download the tool Here.
The source code is Here.

OneNote page xsl transformation

As we saw in a previous post, OneNote API is xml-based. You call the OneNote Application object to obtain almost all needed information as xml strings.

One of that information is the page content. Whose schema is defined as explained in the previous post.

Once you get the page content’s xml string, it is a little bit of work to transform that into a useful html page

To do this, I used an xslt style sheet which is the subject of the current post.

How does it work?

Assume you have an xsl sheet string and the xml string of a page. You can then process both in a way similar to the following code:

 

using System.Xml.Xsl;

 

public static string XmlToHtml(string xmlString, string xslString)
{
    string            html;
    XslCompiledTransform    transform  = null;
    XmlReader         xslReader        = null,
                      xmlReader        = null;
    MemoryStream      memStream        = null;
    StreamReader      sr               = null;
    MemoryStream      xslStream        = null,
                      xmlStream        = null;
    byte[]            xslBytes         = Encoding.UTF8.GetBytes( xslString),
                      xmlBytes         = Encoding.UTF8.GetBytes( xmlString);

    xslStream    = new MemoryStream( xslBytes);
    xmlStream    = new MemoryStream( xmlBytes);

    transform    = new XslCompiledTransform();
    xslReader    = XmlReader.Create( xslStream);
    xmlReader    = XmlReader.Create( xmlStream);
    memStream    = new MemoryStream();

    transform.Load( xslReader);
    transform.Transform( xmlReader, null, memStream);

    memStream.Position    = 0;

    sr      = new StreamReader( memStream);
    html    = sr.ReadToEnd();
               
    xslStream.Close();
    xmlStream.Close();
    memStream.Close();
    sr.Close();
    xslReader.Close();

    return html;
}

 

 

 

How to get the page xml content?

Assuming you have the page ID, here is a sample code to query the page content through OneNote API:

 

public static string GetPageContentXmlString(string pageId)
{
    var         onenoteApp  = new Application();
    string      pageXml     = null;
    PageInfo    pageInfo    = PageInfo.piAll;

    onenoteApp.GetPageContent(pageId, out pageXml, pageInfo);
    return pageXml;
}

 

Reminder: The page content xsd schema

 

The xsl style sheet general structure

Let us use the iXml explorer to navigate through the xsl stylesheet used to transform the page’s xml into html.

(I searched for ‘match’ to locate templates defined in the stylesheet).

As you may notice, we have an xsl template to handle each OneNote page defined xsd type:

  • A template to process the page root information
  • A template to process the page’s title
  • A template to process outline elements
  • A template to process OEChildren collection
  • … and so forth

Sample xsl code

Process OneNote Element (one:OE)

  <!--
  ************************************************
  one:OE
  ************************************************
  -->
  <xsl:template match="one:OE">
    <xsl:param name="nest_level" select="0" />

    <xsl:variable name="listNode" select="./one:List" />
    <xsl:variable name="quickStyleIndex" select="./@quickStyleIndex" />
    <xsl:variable name ="styleNode" select="msxsl:node-set($quickStyleList)/quickStyle[@index=$quickStyleIndex]" />

    <xsl:variable name="quickStyle">
      <xsl:choose>
        <xsl:when test="$styleNode">
          <xsl:value-of select="$styleNode/@style"/>
        </xsl:when>
      </xsl:choose>
    </xsl:variable>

    <!-- is there any list here? -->
    <xsl:choose>
      <xsl:when test="$listNode">
        <xsl:variable name="number"    select="$listNode/one:Number" />
        <xsl:variable name="txt"      select="./one:T" />
        <xsl:variable name="listItemTag">
          <xsl:text>li</xsl:text>
        </xsl:variable>

        <!-- list tag: either <ol> or <ul> -->
        <xsl:variable name="listTag">
          <xsl:choose>
            <xsl:when test="$number">
              <xsl:text>ol</xsl:text>
            </xsl:when>
            <xsl:otherwise>
              <xsl:text>ul</xsl:text>
            </xsl:otherwise>
          </xsl:choose>
        </xsl:variable>

        <xsl:choose>
          <!-- numbered list? output <ol> -->
          <xsl:when test="$number">
            <xsl:variable name="txtNum"        select="number($number/@text)" />
            <xsl:variable name="fontNum"      select="$number/@font" />
            <xsl:variable name="tag" select="concat('<', $listTag, '>')" />
            <xsl:value-of disable-output-escaping="yes" select="$tag"/>

            <!-- *********** output <li value="xx"> ' style="font-family:', $fontNum, ';"',************* -->
            <xsl:value-of disable-output-escaping="yes" select="concat('<', $listItemTag, ' value=', '"', $txtNum, '">')" />

            <xsl:call-template name="outputListItem">
              <xsl:with-param name="listNode" select="$listNode" />
              <xsl:with-param name="itemText" select="$txt" />
            </xsl:call-template>
          </xsl:when>

          <!-- bullet ? output <ul> -->
          <xsl:otherwise>
            <xsl:variable name="tag" select="concat('<', $listTag, '>')" />
            <xsl:value-of disable-output-escaping="yes" select="$tag"/>
            <!-- *********** output <li> ************* -->
            <xsl:value-of disable-output-escaping="yes" select="concat('<', $listItemTag,'>')" />

            <xsl:call-template name="outputListItem">
              <xsl:with-param name="listNode" select="$listNode" />
              <xsl:with-param name="itemText" select="$txt" />
            </xsl:call-template>
          </xsl:otherwise>

        </xsl:choose>

        <!-- process list's sub items -->
        <xsl:apply-templates select="./one:OEChildren">
          <xsl:with-param name="nest_level" select="1 + $nest_level" />
        </xsl:apply-templates>

        <xsl:apply-templates select="./one:Table" />

        <!-- close the list item tag -->
        <xsl:value-of disable-output-escaping="yes" select="concat('</', $listItemTag,'>')" />
        <!-- close the list tag -->
        <xsl:value-of disable-output-escaping="yes" select="concat('</', $listTag, '>')"/>
      </xsl:when>

      <!-- no list: process all sub items -->
      <xsl:otherwise>
        <xsl:variable name="style0" >
          <xsl:call-template name="string-replace-all">
            <xsl:with-param name="text" select="./@style" />
            <xsl:with-param name="replace" select="''" />
            <xsl:with-param name="by" select="''" />
          </xsl:call-template>
        </xsl:variable>

        <xsl:variable name="alignment" select="./@alignment" />

        <xsl:variable name="style">
          <xsl:choose>
            <xsl:when test="$alignment='right'">
              <xsl:value-of select="concat('width:100%; float:right; display:inline; text-align:right;', $style0)"/>
            </xsl:when>
            <xsl:otherwise>
              <xsl:value-of select="$style0"/>
            </xsl:otherwise>
          </xsl:choose>
        </xsl:variable>

        <xsl:choose>
          <xsl:when test="string-length($style)>0">
            <span style="{$style}">
              <xsl:apply-templates />
            </span>
          </xsl:when>

          <!-- ****** no style defined -->
          <xsl:otherwise>
            <xsl:choose>
              <xsl:when test="$quickStyle">
                <span style="{$quickStyle}">
                  <xsl:apply-templates>
                    <xsl:with-param name="nest_level" select="1 + $nest_level" />
                  </xsl:apply-templates>
                </span>
              </xsl:when>

              <xsl:otherwise>
                <span>
                  <xsl:apply-templates>
                    <xsl:with-param name="nest_level" select="1 + $nest_level" />
                  </xsl:apply-templates>
                </span>
              </xsl:otherwise>
            </xsl:choose>
          </xsl:otherwise>
          <!-- ****** end of : no style defined -->
        </xsl:choose>

      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>

Output a list (either numbered or bulleted)

<!--
  ************************************************
  output list item (numbered / bulleted)
  ************************************************
  -->
  <xsl:template name="outputListItem" match="one:List">
    <xsl:param name="listNode" />
    <xsl:param name="itemText" />

    <xsl:variable name="number"        select="$listNode/one:Number" />
    <xsl:variable name="bullet"        select="$listNode/one:Bullet" />
    <xsl:variable name="txtNum"        select="number($number/@text)" />

    <xsl:choose>
      <xsl:when test="$number">
        <xsl:value-of select="normalize-space($itemText)" disable-output-escaping="yes"/>
      </xsl:when>

      <xsl:otherwise>
        <xsl:value-of select="normalize-space($itemText)" disable-output-escaping="yes"/>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>

 

Process page images

In the page’s xml, images are returned as base64 strings.

To process an image into html, the xsl template looks like the following:

  <!--
  ************************************************
  one:Image
  ************************************************
  -->
  <xsl:template match="one:Image">
    <xsl:variable name="imgWidth"select="substring-before( number(./one:Size/@width) * 1.33, '.')"/>
    <xsl:variable name="imgHeight" select="substring-before( number(./one:Size/@height) * 1.33, '.')"/>
    <xsl:variable name="imgData"    select="./one:Data" />
    <xsl:variable name="oneFormat"  select="./@format" />
    <xsl:variable name="htmlformat">
      <xsl:choose>
        <xsl:when test="$oneFormat='png'">
          <xsl:text>data:image/png;base64</xsl:text>
        </xsl:when>
        <xsl:otherwise>
          <xsl:text>data:image/jpg;base64</xsl:text>
        </xsl:otherwise>
      </xsl:choose>
    </xsl:variable>

    <xsl:variable name="htmlWidth">
      <xsl:choose>
        <xsl:when test="$imgWidth">
          <xsl:value-of select="$imgWidth"/>
        </xsl:when>
        <xsl:otherwise>
          <xsl:text>96%</xsl:text>
        </xsl:otherwise>
      </xsl:choose>
    </xsl:variable>

    <xsl:variable name="htmlHeight">
      <xsl:choose>
        <xsl:when test="$imgHeight">
          <xsl:value-of select="$imgHeight"/>
        </xsl:when>
        <xsl:otherwise>
          <xsl:text>auto</xsl:text>
        </xsl:otherwise>
      </xsl:choose>
    </xsl:variable>


    <img width="{$htmlWidth}" height="{$htmlHeight}" src="{$htmlformat}, {$imgData}"/>
  </xsl:template>

 

Download the xsl stylesheet

You may download the entire xsl stylesheet (for OneNote 2013 and 2016) here

Json object explorer

[json objects =>to property bags =>to objects]

Reducing dependency between clients and services is a major common question in software solutions.

One important area of client/server dependencies lies in the structure of objects involved in exchanged messages (requests / responses). For instance: a new property inserted to an object on service side, often crashes the other side (client) until the new property is introduced on the involved object.

I previously posted about loose coupling through property bags abstractions.

My first approach was based on creating a common convention between service and client which implies transforming involved objects into property bags whose values would be assigned as needed to business objects at each side on runtime. That still seems to be a 'best solution' in my point of view.

Another approach is to transform the received objects (at either side: server/client) into property bags before assigning their values to the related objects.

This second approach is better suited for situations where creating a common convention would be difficult to put in place.

While working on some projects based on soap-xml messages, I wrote a simple transformer: [xml => property bags => objects].The transformer then helped write an xml explorer (which actually views xml content as its property bag tree. You can read about this in a previous post).

Another project presented a new challenge in that area, as the service (JEE) was using Json format for its messages. In collaboration with the Java colleagues, we could implement the property bag approach which helped ease client / server versioning issues.

A visual tool, similar to xml explorer, was needed for developers to explore json messages' structures. And that was time for me to write a new json <==>-property bag parser.

The goal was to:

  • Transform json content to property bags
  • Display the transformed property bag tree

Using Newtonsoft's Json library – notably its Linq extensions – was essential.

Hereafter the global dependency diagram of the Json explorer app:

 

The main method in the transformation is ParseJsonString to which you provide the string to be parsed.

Its logic is rather simple:

  • A json string is the representation of either:
    • An object:
      • Read its properties (which may contain arrays… see below)
    • Or an array of:
      • Objects: read the array's objects
      • Arrays: read the array's arrays

 

Code snippets

The parse json string method

public static PropertyBag ParseJsonString(string jsonString)
{
    JObject jObj = null;
    PropertyBag bag = new PropertyBag("Json");
    ObjProperty bagRootNode;
    JArray jArray = null;
    string exceptionString = "";

    /// the json string is either:
    /// * a json object
    /// * a json array
    /// * or an invalid string

    // try to parse the string as a JsonObject
    try
    {
        jObj = JObject.Parse(jsonString);
    }
    catch (Exception ex)
    {
        jObj = null;
        exceptionString = ex.Message;
    }

    // try to parse the string as a JArray
    if(jObj == null)
    {
        try
        {
            jArray = JArray.Parse(jsonString);
        }
        catch (Exception ex2)
        {
            jArray = null;
            exceptionString = ex2.Message;
        }

        if(jArray == null)
        {
            bag.Add(new ObjProperty(_exceptionString, null, false) { ValueAsString = exceptionString });
            return bag;
        }
    }

    bagRootNode = new ObjProperty("JsonRoot", null, false);

    if(bagRootNode.Children == null)
        bagRootNode.Children = new PropertyBag();

    bag.Add(bagRootNode);

    if(jObj != null)
    {
        bagRootNode.SourceDataType = typeof(JObject);
        bagRootNode.Children = ParseJsonObject(bagRootNode, jObj);
    }
    else if(jArray != null)
    {
        bagRootNode.SourceDataType = typeof(JArray);
        bagRootNode.Children = ParseJsonArray(bagRootNode, jArray);
    }

    return bag;
}

 

Parse json array code snippet

 

private static PropertyBag ParseJsonArray(ObjProperty parentItem, JArray jArray)
{
    if(parentItem == null || jArray == null)
        return null;

    ObjProperty childItem;

    if(parentItem.Children == null)
        parentItem.Children = new PropertyBag();

    PropertyBag curBag        = parentItem.Children;

    foreach(var item in jArray.Children())
    {
        JObject jo     = item as JObject;
        JArray subArray = item as JArray;
        PropertyBag childBag;
        Type nodeType = subArray != null ? typeof(JArray) : typeof(JObject);

        childItem    = new ObjProperty("item", parentItem, false) { SourceDataType = nodeType };

        if (jo != null)
            childBag = ParseJsonObject(childItem, jo);
        else if(subArray != null)
            childBag    = ParseJsonArray(childItem, subArray);
        else
            continue;

        curBag.Add(childItem);
    }

    return curBag;
}

 

Json to Xml

As, now, we have the json content in property bags, we can almost directly get the xml equivalent (see screenshot below).

The used sample json string: for the following screenshot:

 

{
    "web-app": {
    "servlet": [
    {
        "servlet-name": "cofaxCDS",
        "servlet-class": "org.cofax.cds.CDSServlet",
        "init-param": {
            "configGlossary:installationAt": "Philadelphia, PA",
            "configGlossary:adminEmail": "ksm@pobox.com",
            "configGlossary:poweredBy": "Cofax",
            …
            …
            "maxUrlLength": 500
            }
    },
    {
        "servlet-name": "cofaxEmail",
        "servlet-class": "org.cofax.cds.EmailServlet",
        "init-param": {
            "mailHost": "mail1",
            "mailHostOverride": "mail2"
            }
    },
    {
        "servlet-name": "cofaxAdmin",
        "servlet-class": "org.cofax.cds.AdminServlet"
    },

    {
        "servlet-name": "fileServlet",
        "servlet-class": "org.cofax.cds.FileServlet"
    },
    {
        "servlet-name": "cofaxTools",
        "servlet-class": "org.cofax.cms.CofaxToolsServlet",
        "init-param": {
            "templatePath": "toolstemplates/",
            "log": 1,
            …
            …
            "adminGroupID": 4,
            "betaServer": true
            }
        }
    ],
    "servlet-mapping": {
        "cofaxCDS": "/",
        "cofaxEmail": "/cofaxutil/aemail/*",
        "cofaxAdmin": "/admin/*",
        "fileServlet": "/static/*",
        "cofaxTools": "/tools/*"
        },

    "taglib": {
        "taglib-uri": "cofax.tld",
        "taglib-location": "/WEB-INF/tlds/cofax.tld"
        }
    }
}



Screenshot

 

You may download the binaries here!

The source code is available here!

OneNote Explorer

In January 2004, Chris Pratley wrote about "OneNote genesis". His article ended by this sentence: "…I can see how this might become addictive."

Yes, as many people who know OneNote often use the word, "addictive" is a correct adjective for OneNote.

What makes it addictive is probably the fact that it is a medium for 'not-yet-documented' ideas. In the same time, it offers a good and simple hierarchical storage that helps organize those ideas for future documentation.

One great thing is that OneNote exposes its objects and methods for developers through an API for extending its features. There are many feature-rich add-ins for OneNote, which can fit your needs in several areas.

In my case, I needed a sort of 'periscope' to explore my own notes. A tool that can let me see my notes ordered by creation or last update dates, mark and retrieve some of them as favorite items, have a quick preview of a note, locate the section's file folder, search notes' titles and/or content… etc.

Using the API, I could write a 'OneNote Explorer'… a tool I started writing in 2010, enriching it with new features from time to time.

OneNote API

OneNote API is xml-based. A set of methods in the Application Interface let you get information about opened notebooks and their structures (section groups, sections, pages… etc.) in xml format.
Understanding OneNote xsd schema is thus essential.

OneNote XSD overview: main objects

 onenote xsd

  • A Notebook is (similar to file folder) a sequence of:
    • Sections
    • And/or Section groups. Where a Section group itself is a sequence of:
      • Section groups
      • And/or Sections. Where a Section is a sequence of:
        • Pages.

OneNote Page definition

onenote page xsd

  • Apart from its attributes (see xsd elements above), a Page is a set of either:
    • An Image
    • A Drawing
    • A File
    • A Media
    • An Outline (similar to html main <div>) which is (somewhat simplified here) is a sequence of:
      • OE children (OneNote Elements). Each OE can be either:
        • An Image
        • A Table
        • Drawing
        • Or a sequence of:
          • T (text range)

Application overview

Application view model classes:

 application main classes

 

A static class (OneNoteHelpers) exposes several methods to communicate with OneNote API and create / update the view model objects as required:

 code map 1

Summary of methods exposed by the helper static class:

 onenote helpers

Features

  • View selected section pages. Search titles, sort the datagrid, add page to favorites, open page in OneNote, html preview

feature 1

  • View all notebook pages. Search titles, sort the datagrid, add page to favorites, open page in OneNote, html preview

feature 2

  • Search selected notebooks

feature 3

  • Manage favorites: delete, preview, open in OneNote…

feature 4

Page preview note

The application's Preview button displays the html content of the selected page. As OneNote API can return the page xml content, an xslt style sheet (with templates per each element type of the xsd definition) allows a simple and quick preview.

sample page 

The simple page's xml tree

sample page's xsd

The page xml code

 
<?xml version="1.0" encoding="utf-8"?>
<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2013/onenote"
         ID="{138E40BA-13BB-4D24-A78C-D92E4E23D574}{1}{E1949424590587215702781963951539196781170461}"
         name="Sample page title" dateTime="2018-04-20T18:53:38.000Z"
         lastModifiedTime="2018-04-20T19:00:30.000Z"
         pageLevel="1"
         isCurrentlyViewed="true"
         selected="partial"
         lang="en-US">
<!-- ***************** quick styles ***************************** -->
<one:QuickStyleDef index="0"
                     name="PageTitle"
                     fontColor="automatic"
                     highlightColor="automatic"
                     font="Calibri Light"
                     fontSize="20.0" spaceBefore="0.0" spaceAfter="0.0" />
 
<one:QuickStyleDef index="1"
                     name="p"
                     fontColor="automatic"
                     highlightColor="automatic"
                     font="Calibri"
                     fontSize="12.0"
                     spaceBefore="0.0" spaceAfter="0.0" />
    
<!-- ********** page settings ********** -->
<one:PageSettings RTL="false" color="automatic">
<one:PageSize>
<one:Automatic />
</one:PageSize>
<one:RuleLines visible="false" />
</one:PageSettings>
    
<!-- ********** title ********** -->
<one:Title selected="partial" lang="en-US">
<one:OE author="taoffi" authorInitials="T.N."
            lastModifiedBy="taoffi"
            lastModifiedByInitials="T.N."
            creationTime="2018-04-20T18:53:46.000Z"
            lastModifiedTime="2018-04-20T18:53:46.000Z"
            objectID="{9DB0692F-3758-49BC-8E87-F040F40599C2}{15}{B0}"
            alignment="left"
            quickStyleIndex="0" selected="partial">
<one:T><![CDATA[Sample page title]]></one:T>
<one:T selected="all"><![CDATA[]]></one:T>
</one:OE>
</one:Title>
    
<!-- ********** page content (main <div>) ********** -->
<one:Outline author="taoffi"
             authorInitials="T.N."
             lastModifiedBy="taoffi"
             lastModifiedByInitials="T.N."
             lastModifiedTime="2018-04-20T19:00:28.000Z"
             objectID="{9DB0692F-3758-49BC-8E87-F040F40599C2}{30}{B0}">
<one:Position x="36.0" y="86.4000015258789" z="0" />
<one:Size width="123.9317169189453" height="14.64842319488525" />
<one:OEChildren>
        <!-- ********** paragra^ph ********** -->
<one:OE creationTime="2018-04-20T18:53:47.000Z"
             lastModifiedTime="2018-04-20T18:53:52.000Z"
             objectID="{9DB0692F-3758-49BC-8E87-F040F40599C2}{33}{B0}"
             alignment="left"
             quickStyleIndex="1">
         <!-- ********** paragraph text ********** -->
<one:T><![CDATA[Sample page text]]></one:T>
</one:OE>
</one:OEChildren>
</one:Outline>
</one:Page>

 

OneNote 2010 vs. 2013 and above

There are some compatibility issues between OneNote API for 2010 version and 2013 and above. More changes have been introduced in 365 version.

The downloadable binaries here are for OneNote 2013 and 2016 desktop.

You may download the binaries Here!

Back to earth – Xamarin.Forms Cuvée 2017

After some months spent on other project types, so glad to find back Xamarin.Forms and to see how it evolved (matured) with the integration in Visual Studio 2017.

The new Xamarin.Forms project template is awesome and pedagogic. You can start building an entire professional solution around the basic objects delivered within that template!

This 2017 spring (well: it is autumn elsewhere!) other interesting projects also blossomed. Don't miss Grial 2.0 of UXDivers. A great ergonomic framework for Xamarin.Forms.

Those guys @UXDivers are unique! (Will talk about this later)

The new Xamarin.Forms template base objects

Inside the main portable project, you will find the 'Helpers' folder containing some interesting foundation objects, of which, the ObservableObject class.

ObservableObject is the base class of another delivered sample object 'BaseDataObject' of which derives the 'Item' object… another delivered sample object.

 

The relationship between those objects are shown in the following class diagram:

 

That looks to be a great and mature foundation. Deriving your objects from this simple architecture would allow you, among other benefits, to have a reliable property change notification mechanism (See my last year's post about this question)

 

Great… But!

As often among our developer community, we may agree on an architectural pattern and disagree on parts of its implementation.

In the new Xamarin implementation, a class calls its parent to set a property value. The parent then sets the new value and notifies the property change.

 

Here is the base class's (ObservableObject) code to set a property value:

 

protected bool SetProperty<T>(ref T backingStore, T value,
             [CallerMemberName]string propertyName = "",
             Action onChanged = null)
{
     if (EqualityComparer<T>.Default.Equals(backingStore, value))
         return false;

      backingStore = value;
     onChanged?.Invoke();
     OnPropertyChanged(propertyName);
     return true;
 }

 

 

A derived class can then use this:

 

string description = string.Empty;

public string Description
{
     get { return description; }
     set { SetProperty(ref description, value); }
}

 

 

In the above example, the property change notification is executed using the caller name (thanks to CompilerService.CallerMemberName attribute) unless the name is explicitly specified.

 

Note: msdn documentation

An Empty value or null for the propertyName parameter indicates that all of the properties have changed.

 

In so many case, a property change may require notifying the change of one or more of other object properties. A simple example would be: when you change the 'Birth date' of a person object, which requires notifying the change of his or her Age property as well.

In fact, an object's property value is, in most cases, linked to business rules and is better handled at the object's level. The property change notification, as its name suggests, is related to notifying external objects who may be interested by the value change of that property.

Summarizing this need by a SET mechanism can be good but does not seem to be an all-purpose solution.

We still need to complement this with the 'old' notification mechanism (based on Expressions) exposed in the pattern of the abovementioned post (which, itself, derives from other community knowledge as I mentioned!)

 

[This post corrects and clarifies points mentioned in a previous post]

Comparable sets

When we manipulate various collections of items, you often need to compare them.

Functionally speaking, comparing two collections (set1 and set2), in its simplest form, may mean:

  • Get the collection of items in set1 that are not in set2 (or vice versa)
  • Get the collection of identical items
  • Get the collection of different items.

 

In my case, I needed to obtain such compared collections in several projects (Xml files, Open Xml packages, swagger files, ms build project settings… I talked about in previous posts).

Several interfaces that come with .net may be of help in many contexts (think about IComparable, IEquitable… etc.). Though quite efficient for sorting and equality tests, they do not seem to be the right choice for solving this particular need for comparing collections.

 

For a collection to produce what we need (identical items, different items and items not in 'another' collection), its items must be able to provide answers to several basic questions.

For an item to tell if it is 'different' from or 'identical' to another, they must first be 'comparable'.

In the case of an Xml node for instance, we might consider that two nodes having the same name are comparable. They then can be different if their values are different. And can be identical if their values are identical.

Let us assume an object that provides such answers:

 

public interface IComparableItem
{
   bool IsComparable(IComparableItem other);
   bool IsDifferent(IComparableItem other);
   bool IsIdentical(IComparableItem other);
}

 

And a generic collection of such items

public class ComparableSet<T> : List<T> where T: IComparableItem

 

In such collection, we can then extract items matching our needs:

 

// items not in the other collection
public List<T> ItemsNotIn(ComparableSet<T> otherSet)

{
   List<T>      list   = new List<T>();

   if(otherSet == null)
      return list;

   foreach(var item in this)
   {
      // get an item of the other collection that is comparable to this item
      var comparable   = otherSet.FirstOrDefault(i => i.IsComparable(item));

      if(comparable == null)
         list.Add(item);
   }

   return list;
}

 

Sample implementation - The comparable item

 

public class ComparableSetItem : IComparableItem
{
   public string Name      { get; set; }
   public string Value      { get; set; }

   public bool IsIdentical(IComparableItem other)
   {
      ComparableSetItem   obj   = other == null ? null : other as ComparableSetItem;
      if(obj == null)
         return false;

      if(obj == this)
         return true;

      return obj.Name == this.Name && obj.Value == this.Value;
   }

   public bool IsDifferent(IComparableItem other)
   {
      ComparableSetItem   obj   = other == null ? null : other as ComparableSetItem;

      if(obj == null || obj == this)
         return false;

      return obj.Name == this.Name && obj.Value != this.Value;
   }

   public bool IsComparable(IComparableItem other)
   {
      ComparableSetItem   obj   = other == null ? null : other as ComparableSetItem;

      if(obj == null || obj == this)
         return false;

      return obj.Name == this.Name;
   }
}

 

The comparable generic collection

 

public class ComparableSet<T> : List<T> where T: IComparableItem
{
   // items not in the other collection
   public List<T> ItemsNotIn(ComparableSet<T> otherSet)
   {
      List<T>      list   = new List<T>();

      if(otherSet == null)
         return list;

      foreach(var item in this)
      {
         var comparable   = otherSet.FirstOrDefault(i => i.IsComparable(item));

         if(comparable == null)
            list.Add(item);
      }

      return list;
   }

 

 

 

   public List<T> IdenticalItems(ComparableSet<T> otherSet)
   {
      List<T>      list   = new List<T>();

      if(otherSet == null)
         return list;

      foreach(var item in this)
      {
         var identicals   = otherSet.Where(i => i.IsIdentical(item));

         if(identicals == null)
            continue;

         foreach(var itemx in identicals)
            list.Add(itemx);
      }

      return list;
   }

 

   public List<T> DifferentItems(ComparableSet<T> otherSet)
   {
      List<T>      list   = new List<T>();

      if(otherSet == null)
         return list;

      foreach(var item in this)
      {
         var      diffs   = otherSet.Where(i => i.IsComparable(item));

         if(diffs == null)
            continue;

         foreach(var otherItem in diffs)
         {

            if(item.IsDifferent(otherItem))
               list.Add(otherItem);
         }
      }

      return list;
   }
}

 

A simple console method to test this implementation

 

public static void TestComparableSets()
{
   
ComparableSet<ComparableSetItem>   list1   = new ComparableSet<ComparableSetItem>();
   
ComparableSet<ComparableSetItem>   list2   = new ComparableSet<ComparableSetItem>();

   list1.Add(new ComparableSetItem() { Name = "name1",    Value = "value1" });
   list1.Add(new ComparableSetItem() { Name = "name2",    Value = "value2" });
   list1.Add(new ComparableSetItem() { Name = "name3",    Value = "value3" });
   list1.Add(new ComparableSetItem() { Name = "name4",    Value = "value4" });
   list1.Add(new ComparableSetItem() { Name = "name5",    Value = "value5" });

   list2.Add(new ComparableSetItem() { Name = "name1",    Value = "value10" });
   list2.Add(new ComparableSetItem() { Name = "name2",    Value = "value2" });
   list2.Add(new ComparableSetItem() { Name = "name3",    Value = "value30" });
   list2.Add(new ComparableSetItem() { Name = "name30",   Value = "value300" });
   list2.Add(new ComparableSetItem() { Name = "name40",   Value = "value40" });
   list2.Add(new ComparableSetItem() { Name = "name5",    Value = "value5" });
   list2.Add(new ComparableSetItem() { Name = "name50",   Value = "value50" });
   list2.Add(new ComparableSetItem() { Name = "name6",    Value = "value60" });

   var      list1NotIn2 = list1.ItemsNotIn(list2);       // {n4/v4}
   var      list2NotIn1 = list2.ItemsNotIn(list1);       // {n30/v30},  {n40/v40},
                                                         // {n50/v50},  {n6/v60}
   var      list1Diff2  = list1.DifferentItems(list2);   // {n1/v10},   {n3/v30}
   var      list2Diff1  = list2.DifferentItems(list1);   // {n1/v1},    {n3/v3}
   var      ident12     = list1.IdenticalItems(list2);   // {n2/v2},    {n5/v5}
   var      ident21     = list2.IdenticalItems(list1);   // {n2/v2},    {n5/v5}
}

 

Using the strategy pattern

The above implementation assumes that you have control upon the collection items structures and are able to change their code.

If that is not the case (like, for instance, when using objects defined in a third party's library) you may use the strategy design pattern in a way similar to the IComparer interface.

 

public interface ICollectionItemComparer
{
   bool IsComparable(object obj1, object obj2);
   bool IsDifferent(object obj1, object obj2);
   bool IsIdentical(object obj1, object obj2);
}

 

The generic collection may then be defined in a way similar to the following:

 

public class ComparableSet<T> : List<T> where T: class
{
   // sample items not in the other collection using strategy pattern
   public List<T> ItemsNotIn(ComparableSet<T> otherSet, ICollectionItemComparer comparer)
   {
      List<T>      list   = new List<T>();

      if(otherSet == null)
         return list;

      foreach(var item in this)
      {
         var comparable   = otherSet.FirstOrDefault(i => comparer.IsComparable(item, i));

         if(comparable == null)
            list.Add(item);
      }

      return list;
   }

Revisiting the Trie –applied to text search and indexing [.net]

A Trie is an efficient tree structure for storing and searching hierarchically-structured information.

For more detailed information you may have a look Here, Here or Here… to mention a few.

In this article, I take 'Text' as the information to be handled.

Text can be viewed as a tree of 'character' nodes. Each set of these nodes compose a 'Word' building block. Several 'Words' building block may share a same set of character nodes.

Let us look at words like: trie, tree, try, trying. All of them share the 't' and 'r' root nodes. 'try' and 'trying' have their 't', 'r', 'y' as common nodes. This can be presented in the following simple diagram:

Or, in a more graphical presentation, like the figure below (green nodes represent end of words):

 

In the case of Text, the efficiency of a Trie is that any word of a given text will start with one of the known alphabetical characters of the language used. Which represents a relatively limited number of nodes.

To enhance our tree structure, we may choose to compose our root nodes with only characters used in the manipulated text (i.e. dynamically compose our root character dictionary).

Trie classes

The following class diagram presents the main Trie model used in the application code:

 

  • CharDictionaryItem: stores one character information (its 'char' value… from which we can get its int value)
  • CharDictionary: the collection or CharDictionaryItem used as the root dictionary for all nodes
  • TrieNode: a trie node item. It refers to one of the above dictionary's nodes. Contains a IsEndOfWord flag that tells whether the node is the end of a word building bloc. And provides information related to its status and position in the tree (Children, Neighbors, IsExpanded…). Through its tree position and flags, a TrieNode object can provide us with useful information such as 'words' (Strings) found at its specific position.
  • TrieNodeList: a collection of TrieNode items.
  • Trie: is the central object. It contains a CharDictionary and a TrieNodeList. This class is implemented as a singleton in the current sample application.

Collection indexers

Collections (CharDictionary, TrieNodeList) expose a useful indexer that retrieves an item by its character. Those indexers will be used through the code for retrieving items.

TrieNodeList indexer:

 

public TrieNode this[char c]
{
get { return this.FirstOrDefault(item => item.Character == c); }
}

CharDictionary indexer:

 

public CharDictionaryItem this[char c]
{
get { return this.FirstOrDefault( item => item.Character == c); }
}

The sample scenario

The sample application proposes the following usage scenario:

  • User selects a text file
  • Parse the file's contents: get its words blocks
  • Compose the root character dictionary
  • Build the words tree
  • Display the presentation tree + additional search functionalities

Parsing text – sample code

Parse text words (having a minimum length: (= 3 chars in the sample app))

 

int ParseStringTask(string str, int minWordLength)
{
// split text words
string[] words = str.Split(new string[]
{
" ", "\r\n", "(", ")", ",", ".", "'", "\"",
":", "/", "'", "'", "«", "»", "-", "+", "=",
"*", "[", "]", "{", "}", ";", "&", "<",
">", "|", "\\", "`", "^", "…", "\t"    
}, StringSplitOptions.RemoveEmptyEntries);

string word;

foreach (string w in words)
{
word = w.Trim();

if (word.Length >= minWordLength)
{
this.AddString(word); // add this word to the tree
}
}

return this._nodes.Count;
}

AddString method: creates the new dictionary items / adds the tree nodes of a string block (word)

 

public void AddString(string str)
{
char c = str[0];
CharDictionaryItem dicoItem;
TrieNode firstNode = _nodes[c],
node;
int ndx,
len = str.Length;

if(firstNode == null)
{
// find the dictionary item. add it if none found
dicoItem = GetDictionaryItem(c);
firstNode = _nodes.AddTail(dicoItem);
}

for(ndx = 1; ndx < len; ndx++)
{
c = str[ndx];
node = firstNode.Children[c];
// find the dictionary item. add it if none found
dicoItem = GetDictionaryItem(c);

if(node == null)
{
node = firstNode.Children.AddTail(dicoItem);
}

if(ndx >= len -1)
{
// set the End of word flag
node.IsEndOfWord = true;
}

firstNode = node;
}
}

Reading back the trie words

How to find back our parsed words through the trie?

Here is a sample code to find words located at a given node:

TrieNode.Strings property:

 

public List<string> Strings
{
get
{
List<string> list = new List<string>();
var childWords = from c in _children where(c._isEndofWord == true) select c;
List<string> neigbors = Next == null ? null : Next.Strings;
string strThis = this.Character.ToString();

// add words found in immediate neighbors
if(childWords != null)
{
string str = strThis;

foreach(TrieNode n in childWords)
{
str += n.Character.ToString();

if(n.IsEndOfWord)
list.Add( str);
}
}

// add words that may be found in child nodes
foreach(TrieNode childNode in _children)
{
foreach(string str in childNode.Strings)
list.Add(strThis + str);
}

return list;
}
}

 

User interface (WPF)

Trie nodes can easily be presented in a TreeView. According to the trie node flag, its corresponding tree view node should be able to indicate this (through different image in our example, using a Converter)

The Tree view item template may be:

 

<TreeView.ItemTemplate>
<HierarchicalDataTemplate ItemsSource="{Binding Children}">
<StackPanel Orientation="Horizontal" Margin="0" >
<Image Width="14"
Source="{Binding Converter={StaticResource nodeImageConverter}}"
Margin="0,0" />
<TextBlock Text="{Binding Character}" Padding="4,0"
Margin="4,0" VerticalAlignment="Center" />
</StackPanel>
</HierarchicalDataTemplate>
</TreeView.ItemTemplate>

 

Searching trie words

Fins words starting with a given string (Trie class)

 

public List<string> StartsWith(string str)
{
List<String> list = new List<string>();

char c = str[0];
TrieNode node0 = this._nodes[c];

if(node0 == null)
return list;

List<string> strings = node0.Strings;

foreach(string s in strings)
{
if(s.StartsWith(str, true, CultureInfo.CurrentCulture))
list.Add(s);
}
return list;
}

 

Fins words containing a given string (Trie class)

 

public List<string> Containing(string str)
{
List<String> list = new List<string>();

if(string.IsNullOrEmpty(str))
return list;

foreach(TrieNode node in this._nodes)
{
var strings = from sx in node.Strings where( sx.Contains(str)) select sx;

foreach(string s in strings)
{
list.Add(s);
}
}

return list;
}

 

Sample app screenshots

 

The sample code

You may download the source code Here.

Have fun optimizing the code and adding new features!

A MMXVI side walk: roman numerals

Numeral systems are fascinating!

Presenting values in various numeral systems reveals some hidden aspects of these values and sometimes reveals more about our Human History and knowledge evolution.

Roman is one of these systems. (you may have a look here, here, or here)

A few years ago, I wrote a method to convert decimal numbers into roman. That worked well. The customer wanted a converter for numbering paragraphs. Up to 50 would be largely enough, he said. The developer in my head pushed me up to 9999 (sadly, I abandoned here by lack of time)

A few days ago, I saw someone wearing a T-shirt with 'XCIV' logo. And that reminded me that I never wrote the reverse conversion (from roman to decimal).

That was a good occasion to write this. And as I also have a friend who wants to practice C#, that may be a good exercise.

I found back my old code (to discover it was all to be rewritten!)… and I started working on a new version!

Roman numerals. A brief presentation

As you may know, Roman numerals building blocks are:

Roman

Decimal

I

1

V

5

X

10

L

50

C

100

D

500

M

1000

 

Intermediate values (like in the table below, between 1 and 10) are additions and subtractions of these basic building blocks

Roman

Decimal

 

I

1

 

II

2

1 + 1

III

3

1 + 1 + 1

IV

4

5 – 1

V

5

 

VI

6

5 + 1

VII

7

5 + 1 + 1

VIII

8

5 + 1 + 1 + 1

IX

9

10 – 1

X

10

 

 

Going beyond the M (1000) [presentation and entries]

Roman numerals were, at their era, (an evidence) hand written (probably more often engraved on hard stones!).

That surely does not mean the people did not know or need to count beyond the 1000. We better never forget that numbers and mathematics are much older than our own era… and that most of the great advances in this area had been achieved in other older civilizations!.

My problem here is just a presentation question: how can I write roman numbers beyond the M (1000)?

Old traditionalists use quirky figures that do not seem easily writable using a 'keyboard'.

Like here:

Or here:

To make presentation and entries a little easier with our era's keyboards, I decided to combine other units to create building blocks beyond the 1000. Example: To present the 1000 – 9000 sequence, I used 'XM' and 'VXM':

 

"XM",

// 10000

"XMXM",

// 20000

"XMXMXM",

// 30000

"XMVXM",

// 40000

"VXM",

// 50000

"VXMXM",

// 60000

"VXMXMXM",

// 70000

"VXMXMXMXM",

// 80000

"CMXM",

// 90000

 

For the sequence 1m – 9m, I used characters that are not in the traditional roman building blocks: The 'U', 'W' and 'Y':

"U",

// 1000000

"UU",

// 2000000

"UUU",

// 3000000

"UW",

// 4000000

"W",

// 5000000

"WU",

// 6000000

"WUU",

// 7000000

"WUUU",

// 8000000

"UY",

// 9000000

 

Conversion processing units

The conversion manager object (iRomanNumberDictionary in the above figure) stores a list of roman building blocks.

Each building block corresponds to a decimal sequence factor (1, 10, 100, 1000… etc.) and stores the roman elements of this sequence (see the sample tables above). It also stores the roman group level (a vital info as you will see in the code!)

Decimal to roman

Now, to convert a decimal number into its roman presentation, we proceed this way (here, using 28 as an example):

  • Take the leftmost digit of the number (leftmost of 28 = 2)
  • Set the index of the sequence to look in = count of number's digits - 1 (for 28 that is 2 – 1 = 1)
    • Note: this target sequence is composed of: "X", "XX", "XXX", … etc.
  • Find the value of this sequence at the element index = the leftmost digit – 1 (2 – 1 = 1)
  • In our case, that would be "XX" (which is decimal 20)
  • Recurse call with the remaining digits of the number (remaining of "28" = "8")
    • Note: that call should return "VIII" (first sequence at the 7th position)
  • Add the string to the "XX" (that would then be "XXVIII")

 

Roman to decimal

Roman to decimal is a bit trickier!

Let us take the reverse conversion of the example above ("XXVIII") for which the conversion result should be 28.

  • The conversion step (a parameter) should be set to the highest index of our roman sequences (in my app, I used 7 sequence groups whose factors are: 1, 10, 100, 1000, 10000… 1000000)
  • We have to look for the string in all sequences whose group order is <= the current step
  • Do until we find something:
    • In our example: we look for "XXVIII" in all sequences whose group order is <= 7
    • As the string is not found in any sequence, we continue the search using the string minus 1 char "XXVII"… "XXVI"… "XXV"… "XX"
    • "XX" is found in the sequence whose decimal factor = 10 at number zero-index = 1
    • We store the following value in a List<int>: sequence's decimal factor * (number zero-index + 1). Which results in 10 * 2 = 20
  • Recurse call using the remaining string "VIII" and using the sequence group order – 1 as the conversion step. Add the returned number to our List<int>
  • Return the sum of integers of our List<int>

 

To better understand what goes on in the conversion process, I added a conversion history that explains the steps of each conversion.

Here is the processing history for XXVIII (28):

 

Another processing history for a greater roman number MMMXCVIII (3098):

 

You may download the code (WPF) HERE. Have fun extending and enhancing for the better!

Yet another [rather simple] OpenXML package explorer!

As its name implies, OpenXML is XML-centric technology (i.e. which extensively uses XML to describe the meta-model and contents of each and every element of a document).

OpenXML documents themselves (MS Word, Excel, PowerPoint…) are compressed packages that include numerous xml files which describe the document's meta-models and contents.

To explore these items, you can simply rename the document file to .zip and then navigate through this zip archive to view the elements that may be of interest.

After proceeding this way, you of course must not forget to rename the file back to its original extension (.docx, .xlsx… or whatever). When working on an OpenXML project, you obviously have to do this so many times a day, which becomes a somehow daunting task!

Some very good tools exist to explore OpenXML documents' structures… but most of them expose a logical view of the document rather than an xml-centric view. Which is of course very helpful… but does not apply when you just need to see the physical structure and the xml content of meta-models and elements.

Moreover, most of these tools are somehow out dated (2007-2009…) and many simply crash on most of our 'modern era' OSs!

Add some other good reasons to write another explorer:

It is august, it is raining and I don't know what to do this week end (while listening to Maria Joao Pires playing Mozart Sonatas!)

 

The purpose and tools?

What I need is:

  • Open an OpenXML document as a zip archive (without having to rename it to .zip!)
  • Explore the content of the archive (folders / files)
  • Explore / search the xml tree of xml files that may be found there

 

Adding a reference to System.IO.Compression and System.IO.Compression.FileSystem, you can (relatively easily) present the archive tree structure like in the following image:

I created a view model layer to map my needs to the zip archive objects of the System.IO.Compression namespace:

iZipEntryVM: a view model for zip archive entry which offers some useful properties related to my purpose.

iZipEntryVmList: a collection of the above object which offers static methods to create the tree of a zip archive file as well as a 'selected item' property and selection change event.

iOfficeVM: (singleton) entry point ('main view model' in a way) that orchestrates the basic commands and selection change events.

 

XML property bags explorer, again, to the rescue!

I wrote in a previous post about using PropertyBags to explore xml nodes.

The xml explorer objects, methods and controls written during this research are now packaged into an independent library that can be referenced in the current project.

What I have to do is just wire the selection event of an item in the archive tree to the xml explorer objects.

 

That is, when I click an xml file on the zip archive tree, the selection change event is fired. If the selected item is an xml content, it is opened and its stream is sent to the xml explorer object to display the node tree.

 

In practice:

  • On the TreeView control's selection changed event, the SelectedItem of the view model collection (iZipEntryVmList) is set.
  • The collection fires its own Selection changed event.
  • iOfficeVM subscribes to that event and send the entry stream to the xml explorer:

 

_myArchiveList.SelectedItemChanged += _list_SelectedItemChanged

private async void _list_SelectedItemChanged(object sender, iZipEntryVM selectedItem)
{
    ZipArchiveEntry    entry    = selectedItem.ZipEntry;

     if(entry == null)
       return;
 
    if(selectedItem.IsXmlFile)
        await Task.Run(() => OpenXmlFile(selectedItem));
}


 

async void OpenXmlFile(iZipEntryVM entryVm)
{
    ZipArchiveEntry    zentry    = entryVm.ZipEntry;
    var        bagVm    = XmlExplorer.Instance.PropertyBagVM;

    if(bagVm != null)
    {
        Stream            stream;
        await Task.Run(() =>
        {
            stream        = zentry.Open();
            // this will build the xml nodes' property bag tree
            bagVm.SourceStream = stream;
        });
    }
}



 

Apart from presenting the xml nodes' tree, XML explorer offers some other useful features:

  • Search for content values and/or node names
  • View / copy the xml string of a selected node
  • Export (save) the current OpenXML item to a file.

 

And, not less important, I no more have to rename a file to do that…!

Some screenshots

Explorer General View

Sample search results

 

Binaries and source code

You may download the binaries here

You may also download the source code here.