At the time RSI was one of the prime records collection for the law firms representing the tobacco industry.  We collected mainly health records on plaintiffs in the many lawsuits against big tobacco that were all the rage.  Which meant managing thousands of pages of scanned documents.

We also worked with other clients on document management and several off-the-shelf software products.  These products included Summation, Trial Director, and Concordance.   As part of my job I also conducted training for clients in the use of these products (for a time I was a certified Summation instructor).

As I said, we did custom applications for clients.  Concordance included a scripting language (Concordance Programming Language, or CPL) that allowed for even greater flexibility from that product.  In going through some old drives I came across a CPL script that I had saved off for some reason.  Here’s a part of that script:

main()
{
  int hConv; /* Handle to Conversation */
  int NULL;
  text hszService; /* Service Name Handle */
  text hszTopic; /* Topic Name Handle */
  text hszDocName;
  int db, i, Rec;
  /* Make sure the database is open. */
  if (db.documents >= 0) 
  {
    /* Initialize the strings used to communicate with the DDE program. */
    hszService = "DOCDIR";
    hszTopic = "DDIMAGE";
    /* Try to connect, do a WinExec() if it isn't running. */
    /* It will run DOCDIR if it is in the path. */
    if ((hConv = ddeConnect(hszService,hszTopic)) == NULL) 
    {
      /* No connection, execute the display program, it may not be running. */
      /* Then try to connect to the server a second time. */
      spawn("d:\progra~1\Docdir\Docdir.exe", "");
      sleep(3000);
      hConv = ddeConnect(hszService, hszTopic);

    }
    /* Message("switch", 1); */
    switch(db.type[isfield(db, "REC")]) {
      case 'P':
      case 'T':
        hszDocName = "[SHOWIMAGE(" + db->DOCPREFIX + "-0001" + ")]"; 
        beep(450,5); 
        break;
      case 'N':
        hszDocName = "[SHOWIMAGE( " + rep("0",5-len(trim(str(db->REC)))) + str(db->REC) + "-0001" + ")]"; 
        break;
      default: beep(450,5);
        break;
    }

    i = ddeExec(hConv, hszDocName );
  }
  return(hConv);
}

You can see the strong C influence as well as the use of an early Windows inter-process messaging protocol called Dynamic Data Exchange.  The use of a C-style scripting language was a little unusual at the time (in my experience anyway).

In hindsight I wish I had saved more examples of CPL.  Note to self: make sure to keep examples of current projects.

A histogram can be thought of as a collection of buckets or bins, each with a min, a max, a width (max – min), and a depth (the number of values in each bin).

Typically the number of bins is determined on a case-by-case basis by looking at the data.  When we create a histogram by hand (so to speak) that’s relatively easy to do.  But when writing software to create histograms, we need to estimate the optimal bin count.  There are several ways to do that, two which I use are:

, where Scott’s method is used to calculate h, that is

or

, also known as Sturge’s Formula

Sturge’s Formula is most useful for , so in TraxView I use the first method.  Using a Math.NET Accumulator to simplify things, it looks something like this (warning: this is just a snippet, so there is stuff left out):

Accumulator acc1 = new Accumulator();
double width;
_hist = new Histogram();

switch (_method)
{
case MethodConstant.EqualInterval:

    acc1.AddRange(Data.DataSet.Values.ToArray());

    // scott's choice method for bin width
    width = ((3.5 * acc1.Sigma) / (Math.Pow(acc1.Count, .333333333)));

    BinCount = (int)Math.Ceiling((Data.DataSet.Values.Max() - Data.DataSet.Values.Min()) / width);
    for (int i = 0; i < BinCount; i++)
        _hist.Add(new Bucket(Math.Round(lb, 2), Math.Round(lb + width, 2)));
    break;
}
RecalcBinDepth();

How a histogram can be used

A histogram, by definition, shows the distribution of data.  In TraxView, we use a histogram to see if the binning for our thematic shading makes sense.  Here’s an example, based on some very limited test data I have:

And here’s another example, showing the difference that selecting different bin widths makes:

Questions or comments welcome.

I won’t go into any background information on the K-S statistic, for a very nice introduction head over to T.W. Kirkman’s Statistics to Use page.

How it works

The K-S test compares a sample with a theoretic or standard distribution.   If the difference is less than a critical value function for the number of samples the sample is normal.  So,

Let be the cumulative distribution function (c.d.f.) of a standard, normal distribution,

Let the critical value function C(n) be defined as:

, and

Let be the observed cumulative step function , where k is the number of observations < x.

Then the K-S statistic where the right-tail test

and the left-tail test

Finally, if , then the sample is normal.

The Code

So here’s what I came up with.  No warranties are implied or expressed, etc.  You’ll note that the function makes use of the MathNet libraries.  You can get them at http://www.mathdotnet.com/

public bool IsNormal
{
    get
    {
        // This method uses the Kolmogorov-Smirnov test to determine a normal distribution.
        // The level of significance (alpha) used is .05, and the critical values used are from Table 1 of: 
        // The Kolmogorov-Smirnov Test for Goodness of Fit
        // Frank J. Massey, Jr.
        // Journal of the American Statistical Association
        // Vol. 46, No. 253 (Mar., 1951) (pp. 68-78)

        if (DataSet.Count == 0)
            return false;

        List<double> vals = DataSet.Values.ToList();
        Accumulator acc = new Accumulator(vals.ToArray());
        double dmax = double.MinValue;
        double cv = 0;

        MathNet.Numerics.Distributions.NormalDistribution test = new MathNet.Numerics.Distributions.NormalDistribution(acc.Mean, acc.Sigma);

        // the 0 entry is to force the list to be a base 1 index table.
        List<double> cvTable = new List<double>() { 0, .975, .842, .708, .624, .565,
                                                .521, .486, .457, .432, .410,
                                                .391, .375, .361, .349, .338,
                                                .328, .318, .309, .301, .294};

        test.EstimateDistributionParameters(DataSet.Values.ToArray());
        vals.Sort();

        for (int i = 0; i < vals.Count; i++)
        {
            double dr = Math.Abs(((i + 1) / (double)vals.Count) - test.CumulativeDistribution(vals[i]));
            double dl = Math.Abs(test.CumulativeDistribution(vals[i]) - (i / (double)vals.Count));
            dmax = Math.Max(dmax, Math.Max(dl, dr));
        }

        // get critical value and compare to d(N)
        if (vals.Count <= 10)
            cv = cvTable[vals.Count];
        else if (vals.Count > 10)
            cv = 1.36 / Math.Sqrt(vals.Count);

        return (dmax < cv);
    }
}

How it works

Given a set of PointF, and a test distance, the function returns True/False for nearness, as well as the closest point and the index of that point.

Yes, I already have those last two things in my NearestPoint() function, but I liked that kitchen sink approach here (I guess).  Without further delay, here is the routine:

public static bool IsNear(List<PointF> f1, PointF pt1, double checkDist, out PointF nearPoint, out int nearIndex)
{
    double calcDist = f1.Select(n => new { n, distance = Math.Sqrt(Math.Pow((n.X - pt1.X), 2) + System.Math.Pow((n.Y - pt1.Y), 2)) }).OrderBy(p => p.distance).First().distance;
    nearPoint = f1.Select(n => new { n, distance = Math.Sqrt(Math.Pow((n.X - pt1.X), 2) + System.Math.Pow((n.Y - pt1.Y), 2)) }).OrderBy(p => p.distance).First().n;
    nearIndex = f1.IndexOf(nearPoint); 

    if (calcDist <= checkDist)
        return true;

    return false;
}

Questions? Comments? Let me know.

The Extension library I wrote includes a new method - SaveOverlayLayer2() .

How it works

GeoObjects saves the Overlay into a MapInfo TAB file.  SaveOverlayLayer2() builds on that as the TAB format has a very simple format.  Our first step is to do a standard SaveOverlayLayer call.  This will give us the geographic data in a MapInfo .map file.

Next, we create a temp file to hold the .tab file.  This is a simple text file that looks somthing like this very simple example:

!table
!version 300
!charset Neutral
Definition Table
 Type NATIVE Charset "Neutral"
 Fields 4
 ID Char (35) ;
 Column0 Char (35) ;
 Column1 Char (35) ;
 Column2 Char (35) ;

The column definitions will come from our extended attributes and will be written using a .Net StreamWriter object. At the same time we’ll be building our .dat file, which is nothing more than a standard .dbf:

dbfWriter annoAttrib = new dbfWriter();
annoAttrib.SetConnection(fileInfo.DirectoryName);

Dictionary<string, string>.KeyCollection featAttrs = attrDict[firstKey].Keys;
int firstKey = attrDict.Keys.First();

using (TextWriter streamWriter = new StreamWriter(s))
{
  streamWriter.WriteLine("!table");
  streamWriter.WriteLine("!version 300");
  streamWriter.WriteLine("!charset Neutral");
  streamWriter.WriteLine("");
  streamWriter.WriteLine("Definition Table");
  streamWriter.WriteLine("  Type NATIVE Charset \"Neutral\"");
  streamWriter.WriteLine("  Fields " + attrDict[firstKey].Keys.Count.ToString());

  foreach (string key in featAttrs)
  {
    annoAttrib.AddField(key, "VARCHAR", 35);
    streamWriter.WriteLine("    " + key + " Char (35) ;");
  }
}

Once that is done, we’ll close the stream and write the .dat/dbf file.

string shortFileName = "_attemp";
string baseFileName = Path.GetFileNameWithoutExtension(fileInfo.FullName);

annoAttrib.CreateDBF(shortFileName, true);

OverlayLayer overlay = (OverlayLayer)m.GetOverlayLayer(0);
int feat2Count = ((Features)overlay.Features).Count;

for (int i = 1; i <= feat2Count; i++)
{
    Feature2 copyFeat = (Feature2)((Features)overlay.Features).get_Item(i);
    Dictionary<string, string> featAttrs = new Dictionary<string, string>();
    try
    {
        featAttrs = attrDict[copyFeat.Id];

        foreach (KeyValuePair<string, string> kvp2 in featAttrs)
        {
            annoAttrib.SetFieldValues(kvp2.Key, kvp2.Value);
        }
        annoAttrib.WriteRecord();
    }
    catch { }
}

annoAttrib.EndConnection();

Once that is done, I rename the .dbf to a .dat, and we’re done with the MapInfo file.  If we want to create a Shapefile, we use GeoConvert on the saved MapInfo TAB.

What that means is that if you copy an existing feature to the Overlay layer, none of the attributes are copied with it.  That makes updating the existing feature sort of challenging.  To overcome that limitation, we added a CopyAttributes()  method to our GeoObjects extension library.

It is a pretty simple addition, just get the existing features and add them to a Dictionary for that feature ID.  Do do this we need three things: the Dictionary, a SetAttributes() method, and the CopyAttributes() method.  Something like this:

private static Dictionary<int, Dictionary<string, string>> attrDict = new Dictionary<int, Dictionary<string, string>>();
public static bool SetAttribute(this GeoObjectsMapLib.Feature2 f, string s, string s1)
{
    if (!attrDict.ContainsKey(f.Id))
        attrDict.Add(f.Id, new Dictionary<string, string>());

    if (attrDict[f.Id].ContainsKey(s))
        attrDict[f.Id][s] = s1;
    else
        attrDict[f.Id].Add(s, s1);

    return true;
}

public static void CopyAttributes(this Feature2 f2, Feature f)
{
    object attr = null;

    f.GetAttributes(ref attr);
    Object[] attributes = (Object[])attr;

    for (int j = 0; j < attributes.Length; j++)
        f2.SetAttribute(attributes[j].ToString(), attributes[++j].ToString());
}

In use it looks like this:

Feature2 copyFeat;

foreach (Feature feat in (Features)someLayer.Features) 
{
    copyFeat = (Feature2)overlay.CopyFeature(feat);
    copyFeat.CopyAttributes(feat);
    copyFeat.Update(0);
}

In the meantime, here’s a short function I used to determine the nearest point from a collection to a test point.

public static PointF NearestPoint(List<PointF> f1, PointF pt1, out int nearIndex)
{
    PointF nearPoint = f1.Select(n => new { n, distance = Math.Sqrt(Math.Pow((n.X - pt1.X), 2) + System.Math.Pow((n.Y - pt1.Y), 2)) }).OrderBy(p => p.distance).First().n;
    nearIndex = f1.IndexOf(nearPoint);

    return nearPoint;
}

This function uses a Lambda function to create the delegate for the Select() method.  That delegate computes the distance from the test point to each point in the collection, orders the results, and returns the point with the smallest distance.

It’s probably likely that you have never heard of extensions. Here’s the Microsoft definition:

Extension methods enable you to “add” methods to existing types without creating a new derived type, recompiling, or otherwise modifying the original type. Extension methods are a special kind of static method, but they are called as if they were instance methods on the extended type. For client code written in C# and Visual Basic, there is no apparent difference between calling an extension method and the methods that are actually defined in a type.

Quick aside: So why not create derived classes using GeoObjects classes as bases? Long story short – it can’t be done. Seriously, no kidding, word to your mother, etc.

GeoObjects doesn’t include a method for saving features in WKT format. I found that the easiest way to translate features from GeoObjects format to use in other libraries (like the Field Operations Data Model libs) was to get the WKT for the GeoObjects feature. This method will extend the GeoMap object and uses a Feature2 Object (which is a Feature from the Overlay layer) like so: Map1.ToWKT(feat2). To do this I’ll be using SharpMap to do the heavy lifting.

A good rule of thumb when creating extensions is to isolate them in a separate namespace. In my case I created a GeoObjects Extensions namespace. Next we need to create the method definition:

public static string ToWKT(this AxGeoObjectsMapLib.AxBMGeoMap m, Feature2 f)
 {

 }

The first argument is the object being extended – in this case the Interop version of a GeoMap. The second argument is the feature to be converted (as stated above, a Feature2 object). Next I handle the different typs of GeoObjects Geometries. Here’s the code for a polygon:

if (f.GeometryType == GeometryTypeConstants.gtPolygon)
{
  GeoObjectsMapLib.PolygonGeometry pg = (GeoObjectsMapLib.PolygonGeometry)f.Geometry;
  int pgSections = (pg.SectionCount == 0) ? 1 : pg.SectionCount;
  int low = (pg.SectionCount == 0) ? 0 : 1;

  SharpMap.Geometries.MultiPolygon shMulti = new SharpMap.Geometries.MultiPolygon();
  for (int i = low; i < pgSections + 1; i++)
  {
     int vertexCount = pg.VertexCount(i);
     if (vertexCount > 0)
     {
         OlePoint geopt;
         List<SharpMap.Geometries.Point> tempGeoPoints = new List<SharpMap.Geometries.Point>();
         for (int j = 1; j < vertexCount + 1; j++)
         {
             geopt = new OlePointClass();
             geopt = (OlePoint)pg.GetVertex(j, i);
             m.ConvertCoordinate(CoordinateConversionConstants.ccDataspaceToGeodetic, geopt);
             tempGeoPoints.Add(new SharpMap.Geometries.Point(geopt.x, geopt.y));
         }
         geopt = new OlePointClass();
         geopt = (OlePoint)pg.GetVertex(1, i);
         m.ConvertCoordinate(CoordinateConversionConstants.ccDataspaceToGeodetic, geopt);
         tempGeoPoints.Add(new SharpMap.Geometries.Point(geopt.x, geopt.y));

         SharpMap.Geometries.Polygon shGeo = new SharpMap.Geometries.Polygon(new LinearRing(tempGeoPoints));
         shMulti.Polygons.Add(shGeo);
         }
     }
     if (shMulti.Polygons.Count == 1)
         rtnWKT = shMulti.Polygons[0].AsText();
     else
         rtnWKT = shMulti.ToString();
}

Next time: Enhancing the Feature2 object to allow for true editing of existing features.