A long-lost specialty programming language

Back in the mid-1990s I worked for a legal services firm named RSI (or more correctly, Reproduction Services Incorporated).  I was the Applications Development Manager, responsible for managing development for both internal software and custom applications for clients.

At the time RSI was one of the prime records collection for the law firms representing the tobacco industry.  We collected mainly health records on plaintiffs in the many lawsuits against big tobacco that were all the rage.  Which meant managing thousands of pages of scanned documents.

We also worked with other clients on document management and several off-the-shelf software products.  These products included Summation, Trial Director, and Concordance.   As part of my job I also conducted training for clients in the use of these products (for a time I was a certified Summation instructor).

As I said, we did custom applications for clients.  Concordance included a scripting language (Concordance Programming Language, or CPL) that allowed for even greater flexibility from that product.  In going through some old drives I came across a CPL script that I had saved off for some reason.  Here’s a part of that script:

main()
{
  int hConv; /* Handle to Conversation */
  int NULL;
  text hszService; /* Service Name Handle */
  text hszTopic; /* Topic Name Handle */
  text hszDocName;
  int db, i, Rec;
  /* Make sure the database is open. */
  if (db.documents >= 0) 
  {
    /* Initialize the strings used to communicate with the DDE program. */
    hszService = "DOCDIR";
    hszTopic = "DDIMAGE";
    /* Try to connect, do a WinExec() if it isn't running. */
    /* It will run DOCDIR if it is in the path. */
    if ((hConv = ddeConnect(hszService,hszTopic)) == NULL) 
    {
      /* No connection, execute the display program, it may not be running. */
      /* Then try to connect to the server a second time. */
      spawn("d:\progra~1\Docdir\Docdir.exe", "");
      sleep(3000);
      hConv = ddeConnect(hszService, hszTopic);

    }
    /* Message("switch", 1); */
    switch(db.type[isfield(db, "REC")]) {
      case 'P':
      case 'T':
        hszDocName = "[SHOWIMAGE(" + db->DOCPREFIX + "-0001" + ")]"; 
        beep(450,5); 
        break;
      case 'N':
        hszDocName = "[SHOWIMAGE( " + rep("0",5-len(trim(str(db->REC)))) + str(db->REC) + "-0001" + ")]"; 
        break;
      default: beep(450,5);
        break;
    }

    i = ddeExec(hConv, hszDocName );
  }
  return(hConv);
}

You can see the strong C influence as well as the use of an early Windows inter-process messaging protocol called Dynamic Data Exchange.  The use of a C-style scripting language was a little unusual at the time (in my experience anyway).

In hindsight I wish I had saved more examples of CPL.  Note to self: make sure to keep examples of current projects.

Some Brief Notes on Histograms

To start, how about a simple definition: A histogram is a graphical representation of the distribution of data.

A histogram can be thought of as a collection of buckets or bins, each with a min, a max, a width (max – min), and a depth (the number of values in each bin).

Typically the number of bins is determined on a case-by-case basis by looking at the data.  When we create a histogram by hand (so to speak) that’s relatively easy to do.  But when writing software to create histograms, we need to estimate the optimal bin count.  There are several ways to do that, two which I use are:

A Kolmogorov-Smirnov implementation

The Kolmogorov-Smirnov (hereafter K-S) Statistic, allows for a fairly straight-forward test for both normality and uniformity.  Here I’ll concentrate on normality.

I won’t go into any background information on the K-S statistic, for a very nice introduction head over to T.W. Kirkman’s Statistics to Use page.

How it works

An IsNear() method using C#

Earlier I wrote about finding the NearestPoint() in C# – a handy little function.  I also have a companion function – IsNear().  As with the previous function, IsNear() makes use of Lambda functions and is fairly simple in its approach.

How it works

Given a set of PointF, and a test distance, the function returns True/False for nearness, as well as the closest point and the index of that point.

Yes, I already have those last two things in my NearestPoint() function, but I liked that kitchen sink approach here (I guess).  Without further delay, here is the routine:

public static bool IsNear(List<PointF> f1, PointF pt1, double checkDist, out PointF nearPoint, out int nearIndex)
{
    double calcDist = f1.Select(n => new { n, distance = Math.Sqrt(Math.Pow((n.X - pt1.X), 2) + System.Math.Pow((n.Y - pt1.Y), 2)) }).OrderBy(p => p.distance).First().distance;
    nearPoint = f1.Select(n => new { n, distance = Math.Sqrt(Math.Pow((n.X - pt1.X), 2) + System.Math.Pow((n.Y - pt1.Y), 2)) }).OrderBy(p => p.distance).First().n;
    nearIndex = f1.IndexOf(nearPoint); 

    if (calcDist <= checkDist)
        return true;

    return false;
}

Questions? Comments? Let me know.

Saving an edited layer in GeoObjects

So we’ve extended the GeoObjects library, and seen how to edit an existing feature.  Next we’ll need to save those changes.  The GeoObjects SaveOverlayLayer() method does just what it says – it saves the Overlay Layer along with the two attributes that it carries – Id and Name.  Not so good if you are editing a feature with more attributes.

The Extension library I wrote includes a new method - SaveOverlayLayer2() .

How it works

GeoObjects saves the Overlay into a MapInfo TAB file.  SaveOverlayLayer2() builds on that as the TAB format has a very simple format.  Our first step is to do a standard SaveOverlayLayer call.  This will give us the geographic data in a MapInfo .map file.

Next, we create a temp file to hold the .tab file.  This is a simple text file that looks somthing like this very simple example:

!table
!version 300
!charset Neutral
Definition Table
 Type NATIVE Charset "Neutral"
 Fields 4
 ID Char (35) ;
 Column0 Char (35) ;
 Column1 Char (35) ;
 Column2 Char (35) ;

The column definitions will come from our extended attributes and will be written using a .Net StreamWriter object. At the same time we’ll be building our .dat file, which is nothing more than a standard .dbf:

dbfWriter annoAttrib = new dbfWriter();
annoAttrib.SetConnection(fileInfo.DirectoryName);

Dictionary<string, string>.KeyCollection featAttrs = attrDict[firstKey].Keys;
int firstKey = attrDict.Keys.First();

using (TextWriter streamWriter = new StreamWriter(s))
{
  streamWriter.WriteLine("!table");
  streamWriter.WriteLine("!version 300");
  streamWriter.WriteLine("!charset Neutral");
  streamWriter.WriteLine("");
  streamWriter.WriteLine("Definition Table");
  streamWriter.WriteLine("  Type NATIVE Charset \"Neutral\"");
  streamWriter.WriteLine("  Fields " + attrDict[firstKey].Keys.Count.ToString());

  foreach (string key in featAttrs)
  {
    annoAttrib.AddField(key, "VARCHAR", 35);
    streamWriter.WriteLine("    " + key + " Char (35) ;");
  }
}

Once that is done, we’ll close the stream and write the .dat/dbf file.

string shortFileName = "_attemp";
string baseFileName = Path.GetFileNameWithoutExtension(fileInfo.FullName);

annoAttrib.CreateDBF(shortFileName, true);

OverlayLayer overlay = (OverlayLayer)m.GetOverlayLayer(0);
int feat2Count = ((Features)overlay.Features).Count;

for (int i = 1; i <= feat2Count; i++)
{
    Feature2 copyFeat = (Feature2)((Features)overlay.Features).get_Item(i);
    Dictionary<string, string> featAttrs = new Dictionary<string, string>();
    try
    {
        featAttrs = attrDict[copyFeat.Id];

        foreach (KeyValuePair<string, string> kvp2 in featAttrs)
        {
            annoAttrib.SetFieldValues(kvp2.Key, kvp2.Value);
        }
        annoAttrib.WriteRecord();
    }
    catch { }
}

annoAttrib.EndConnection();

Once that is done, I rename the .dbf to a .dat, and we’re done with the MapInfo file.  If we want to create a Shapefile, we use GeoConvert on the saved MapInfo TAB.

Editing Existing Features with GeoObjects

As I wrote earlier, the Blue Marble GeoObjects library is limited in its editing support.  One major limitation is that objects in the Overlay (or drawing) layer have no attributes that are returned via the GetAttribute() or GetAttributes() methods.

What that means is that if you copy an existing feature to the Overlay layer, none of the attributes are copied with it.  That makes updating the existing feature sort of challenging.  To overcome that limitation, we added a CopyAttributes()  method to our GeoObjects extension library.

It is a pretty simple addition, just get the existing features and add them to a Dictionary for that feature ID.  Do do this we need three things: the Dictionary, a SetAttributes() method, and the CopyAttributes() method.  Something like this:

private static Dictionary<int, Dictionary<string, string>> attrDict = new Dictionary<int, Dictionary<string, string>>();
public static bool SetAttribute(this GeoObjectsMapLib.Feature2 f, string s, string s1)
{
    if (!attrDict.ContainsKey(f.Id))
        attrDict.Add(f.Id, new Dictionary<string, string>());

    if (attrDict[f.Id].ContainsKey(s))
        attrDict[f.Id][s] = s1;
    else
        attrDict[f.Id].Add(s, s1);

    return true;
}

public static void CopyAttributes(this Feature2 f2, Feature f)
{
    object attr = null;

    f.GetAttributes(ref attr);
    Object[] attributes = (Object[])attr;

    for (int j = 0; j < attributes.Length; j++)
        f2.SetAttribute(attributes[j].ToString(), attributes[++j].ToString());
}

In use it looks like this:

Feature2 copyFeat;

foreach (Feature feat in (Features)someLayer.Features) 
{
    copyFeat = (Feature2)overlay.CopyFeature(feat);
    copyFeat.CopyAttributes(feat);
    copyFeat.Update(0);
}

Determining the Nearest Point in C#

One feature of TraxView Mapping is the ability to create contour (or isopleth) maps from soil sample data.   Creating those contours is a bit of a challenge, which I’ll write more about at some point.

In the meantime, here’s a short function I used to determine the nearest point from a collection to a test point.

public static PointF NearestPoint(List<PointF> f1, PointF pt1, out int nearIndex)
{
    PointF nearPoint = f1.Select(n => new { n, distance = Math.Sqrt(Math.Pow((n.X - pt1.X), 2) + System.Math.Pow((n.Y - pt1.Y), 2)) }).OrderBy(p => p.distance).First().n;
    nearIndex = f1.IndexOf(nearPoint);

    return nearPoint;
}

This function uses a Lambda function to create the delegate for the Select() method.  That delegate computes the distance from the test point to each point in the collection, orders the results, and returns the point with the smallest distance.