Creating Screen Scrapers with Visual 3270 for Delphi/C++ Builder

 

 

First, what is a screen scraping application or Host Rejuvenation application?  In its simplest form it is the simple process of “scraping” data from a traditional “green screen” or terminal that communicates with a mainframe or host system:

 

 

In the above screen shot, the OpenConnect Emulator has connected to the University of Florida and has a continuous TCP/IP link to the system.  Host Rejuvenation programs will translate this “raw” data from its vanilla form above and convert it into another form, usually HTML.  Even more importantly is the fact there is no install on the host device and as such represents an attractive solution for PC developers using Visual C++, Visual Basic, Delphi, Java, and C++Builder.  Such applications can include:

 

            College Course Registration On-Line

            Vehicle Registration On-Line

            Parking Tickets On-Line

            Single Sign-on Applications

           

As you can see, this list is only a small portion of a large market and can include Government applications, Medical, Commercial, and Education.  For instance, OpenConnect’s Visual 3270 for ISAPI converts “green screens” into pure HTML versions as shown below:

 

 

The above application was written entirely in Object Pascal using Delphi 6.0 and is the definitive thin client application.  The above application can support up to 500 simultaneous users without significant resources on the server that is processing the requests.  Undoubtedly you may be wondering about the architecture behind this system:

 

 

 

 

 

Obviously the most important component is the COM object in the Active Server Page which will communicate with the out of process server. 

 

Building a Simple Out-of-Process Server

 

We will slowly progress towards a multi-threaded COM application that services 1000’s of simultaneous clients, but first we must create a simple Automation Server and more specifically an out of process server.  An out of process server will give your application the necessary requirements to handle thousands of requests and in fact is the recommended way to use Visual 3270 from within Internet Information Server to maintain state.  The first step is to open Delphi and Create a new Application (File | New Application).  Once the application has been created you must add an Automation Object to the application which will do several things, but primarily it will add a COM interface so that methods can be exposed for controlling your Automation Server. 

 

 

At this point jus press OK to continue the process.  You must now name this interface and select the instancing model and threading model.  For help on these items refer to your Borland documentation.

 

 

As can be seen from the above screen shots you should enter the information as shown.  After pressing OK on the last dialog you will have added a type library to your project that will look similar to the next screen shot.

 

It’s a good idea to remember that Project1 is the name of this Object and has an interface named IMyCOMTest and will be used in various formats in your code such as Project1.MyCOMTest.

 

I want to point out immediately that you CANNOT call methods from Unit1 (Form1) directly from an Automation controller; you can only call methods in the type library!  And as such you will now create a method that can be called from an ASP page.  This may seem strange at first, but as you develop more you will realize that the interface is how tow object will communicate and there you can only call methods and use properties that are defined in the interface (type library) which can then communicate natively with their implementations.

 

 

 

In the above dialog I have added a simple method to the Interface portion of the type library.  Notice that the other portion of the type library is a CoClass and is another method for communicating between processes.  We will discuss this later but for now make sure that your dialog looks exactly like the above image. 

 

Once you have made your type library look like the above, press the Refresh Implementation button on the same dialog which will place the necessary interface code into Unit2.pas realizing that Unit1 represents Form1 and Unit2.pas represents your type library implementation.  You MUST update (refresh implementation) your code after each change in the type library.  Close the dialog after you have finished with it. 

 

At this point we must add some implementation code to the newly created Connect method, but in case you didn’t see it the next screen shot shows you what Unit2 should now look like:

 

 

Understand that this is the function that will be called from an automation controller such as an ISAPI extension (.dll) or another application.  Only those methods and properties that are added to the type library can be called.  So how do you tell Form1 to start a 3270 session?  First, you should open up Form1 and add a TN3270 Emulator to the form.  Its best to always start with a Visual 3270 Emulator and then at another point in time replace it with the more light weight ScreenScraper control.  Your Form1 should now look like the following:

 

 

In the above dialog I have merely added a simple TNEmulatorSSL to the form.  Set the properties to the following:

 

           

Now add a function named MyConnect to the Form1 class so that your code looks like the following:

 

unit Unit1;

 

interface

 

uses

  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,

  Dialogs, TN3270SocketSSL;

 

type

  TForm1 = class(TForm)

    TNEmulatorSSL1: TTNEmulatorSSL;

    procedure TNEmulatorSSL1EOS(Sender: TObject);

  private

    { Private declarations }

  public

    ScreenComplete: boolean; //for End of Screens

 

    function MyConnect: boolean;

 

  end;

 

var

  Form1: TForm1;

 

implementation

 

{$R *.dfm}

 

function TForm1.MyConnect: boolean;

begin

        TNEmulatorSSL1.Connect;

        repeat

           Application.ProcessMessages; //wait until first screen is done!

        until (ScreenComplete);

 

        result := true;

end;

 

procedure TForm1.TNEmulatorSSL1EOS(Sender: TObject);

begin

        ScreenComplete := true; //A Screen has returned!

end;

 

end.

 

The IMPORTANT aspect of the above code is to remember that Visual 3270 is asynchronous and as such, events happen without “blocking” methods which can save you significant resources; however, you must remember that once you issue a COMMAND such as Connect() or F1, or F2, or some other 3270 command such as pressing the ENTER key, you must wait until the mainframe responds by sending an EOR (End of Record) or in this case an EOS (End of Screen).

 

It should be noted here, that some mainframes will send an EOS, but the screen is actually not complete.  You should test your emulation’s screen by using the GetScreen() function to read the text of the screen before assuming that a screen actually is complete.  Most mainframes follow the rules, but many don’t so just make sure that you check the screen but this is really out of the control of the Visual 3270 Emulator.

 

The next step is to merely call the MyConnect() function from Unit2, but you must first add Unit1 to the uses clause in the implementation section.  Add the necessary modifications to Unit2.pas so your code looks similar to the following:

 

 

unit Unit2;

 

{$WARN SYMBOL_PLATFORM OFF}

 

interface

 

uses

  ComObj, ActiveX, Project1_TLB, StdVcl;

 

type

  TMyCOMTest = class(TAutoObject, IMyCOMTest)

  protected

    procedure Connect; safecall;

    { Protected declarations }

  end;

 

implementation

 

uses ComServ, Unit1;

 

procedure TMyCOMTest.Connect;

var

Connected: boolean;

Begin
    Form1.ScreenComplete := false;

    Connected := Form1.MyConnect;

 

end;

 

initialization

  TAutoObjectFactory.Create(ComServer, TMyCOMTest, Class_MyCOMTest,

    ciMultiInstance, tmApartment);

end.

 

 

In the above code, we have added Unit1 to the uses clause in the implementation section and then set the public variable ScreenComplete to false and then merely called the MyConnect() function in Form1.  Pretty straight forward, but notice that we can not call MyConnect() directly from an Automation Controller such as an ASP page!  You should also note that if you do not add the code in Unit1 that repeats until ScreenComplete is true, the Connect() function will return prior to the screen actually being complete, you definitely don’t want this, so remember to return a function only when a screen has completed.  This holds true for issuing any other commands including F1, F2, Clear and others.

 

If all has gone well you should be able to compile the application and then run the application.  In order to register the COM server you must run the application once.  At this point the Operating System is aware of this COM server and can be imported into any COM supporting development environment including Delphi.  Make sure you save your application with the default names and build the application as well.

 

Creating an Automation Controller to Control the Automation Server

 

The next part will demonstrate how to control the Automation server (out of process) that we just created in the previous section.  I like to either do View | Project Manager and add a new Application to the project manager or just completely start a new Delphi session and create a new application.  The choice is yours.

 

Once you have set up your environment, create a new Application and save it as Project2 or anything else, just as long as it isn’t the same name as the Automation Server you created earlier.  The next thing to do is to import the type library that you created in the previous section.  You may not have noticed it in the previous section but there is actually another file associated with Uni2.pas and that was Project1_TLB.pas which is the text form of the type library you graphically entered the Connect() method into.  This file will get recreated with each import into an application setting and as such, if you modify your type library’s interface (not the code) you must re-import this file into each application that uses it, so make sure you plan your COM interface very well.

 

First you must Import the Type library created earlier (our Automation Server) into the new application by selecting Project | Import Type Library from the menu bar.  You should get the following dialog.  Make sure you select the correct type library:

 

 

Press the Create Unit button on this dialog, which will create a Project1_TLB file and place it into the Delphi6\Imports directory.  You can now add this file to the end of the uses clause in the interface section of Unit1.pas.

 

After you have added the type library add two TButtons to Form1.  One named StartBtn with a caption of “Start Server”, and one named ConnectBtn with a caption of “Connect.”

 

 

Form1 should now look like the above.  Add an OnClick event to each button so that your code looks exactly like the next code segment:

 

unit Unit1;

 

interface

 

uses

  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,

  Dialogs, Project1_TLB, StdCtrls;

 

type

  TForm1 = class(TForm)

    StartBtn: TButton;

    ConnectBtn: TButton;

    procedure StartBtnClick(Sender: TObject);

    procedure ConnectBtnClick(Sender: TObject);

  private

    FIntf: IMyCOMTest;

  public

    { Public declarations }

  end;

 

var

  Form1: TForm1;

 

implementation

 

{$R *.dfm}

 

procedure TForm1.StartBtnClick(Sender: TObject);

begin

        Fintf := CoMyCOMTest.Create;

end;

 

procedure TForm1.ConnectBtnClick(Sender: TObject);

begin

        FIntf.Connect;

        ShowMessage('Connected!');

end;

 

end.

 

 

Notice our use of FIntf public variable in the StartBtnClick() procedure.  This will actually fire up our Automation Server and then the ConnectBtnClick() procedure will tell the TNEmulatorSSL1 on the Automation form to connect to its server. 

 

Build the application and give it a test run.  First press the Start Server button and after the Automation server fires up, press the Connect button and see if your emulator connects!

 

 

You should see something similar to the above if all has gone well!  If not check your imports and uses clauses and make sure your code matches all the code as provided in this article.

 

Expanding on this idea only requires that you create a different type of Automation Controller such as ISAPI or ASP Controller.

 

Deciding on ISAPI or ASP

You may be asking yourself which is the best methodology?  ASP offers significant savings on HTML design time but unless you are using advanced MTS methods each connection is instantiated for each web page, which means you can’t really keep state.  This may be exactly what you want, you want to go in, get some data and come back out in which case you would create an Automation Server (in process or out of process) and provide something very similar to the sample we provided earlier.  This has many advantages with the most obvious being the lack of state management required to put your application on the net and the other is a great way to design HTML web pages.

 

ISAPI offers significant performance gains and state management opportunities, but lacks the great HTML development effort that ASP can provide.  The obvious advantage is the ability to load an out of process server at the start of the ISAPI Extension and then for each instance of the extension you can access the already running out of process server. 

 

The choice really comes down to your requirements.  If you need a simple one time access to the mainframe, such as checking on a person’s vehicle registration number, or hospital record, then by all means use an ASP COM combination.  If you require a stateful connection which needs to be managed then consider the ISAPI solution.  There is one more solution that can give you an opportunity to keep stateful connections in either ASP or ISAPI and at the same time manage the individual TN3270 connections and is the choice OpenConnect has used many times and that is a sockets implementation.

 

Deciding on Sockets

If you’re like us, you want to manage state and the TN3270 components without giving up anything.  This is the perfect opportunity for using Sockets.  Using sockets in an ISAPI application requires an out of process executable and what we mean by out of process is a completely independent application that houses all of the components.  This executable operates as a Socket Server for ISAPI applications or ASP applications and what provides the best solution is: the ability to keep the application up and running without bringing down IIS and if required to start the application over without restarting IIS.  The approach is simple, an ISAPI application creates a Socket connection to the Socket Server passing in a new connection request or an existing connection ID.  The Socket Server determines if this is a new connection and starts a new TN3270 session returning the screen data when the OnEOS fires for this Sender.  If the request is an existing connection, then the Socket Server goes to the ID of the session (passed in from the client page) and send the command to the TN3270 session, awaits for OnEOS and returns the data. 

 

What I have liked most about this approach is the ability to start several hundred applications and keep them at a certain location in the TN3270 session, say screen 20 by manually entering commands or by using a macro script.  Then when a request comes in the session doesn’t have to travel from the first screen all the way to the 20th screen, it merely goes to an available session and grabs the data. 

 

Using ASP for down and dirty screen scraping

By far the easiest application is the single use application, where you need to access a mainframe and grab some data and come back.  This is the ideal application for the internet as this represents the classic stateless connection.  For example, we are going to go into the Library and University Information System (LUIS) and find if Star Wars is available for check out at one of the 10 universities in the state of Florida. 

 

First, close all projects in Delphi and then select File | New | Other … and select an ActiveX Library.  This will create the Dynamic Link Library Framework.  Then select File | New | Other and Active Server Object.  If you don’t have Active Server Object, our Visual 3270 Documentation discusses how to create Active Server Pages without using Delphi’s automated wizards.

When you add an Active Server Object you should get the following dialog:

 

 

Make sure your dialog looks like the above and then press OK to continue.  For more information on this dialog and others please review your Delphi/C++ Developer’s guides.  Like before, we are now shown a type library dialog.  We are going to add a very simple method named ConnectMe with no parameters and one property named MyScreen with a value of BSTR.  Make sure your type library looks like the following:

 

Realize also that Delphi subscribes to Get/Set methods for properties and as such you will see two Screen properties in your dialog (one that “sets” and one that “gets”).  Make sure you press the refresh implementation button prior to leaving this dialog.  You will notice that Delphi has created two methods for the Screen property and as such you must fill in the implementation detail for this property.  It is customary that whenever you have a published property, such as Screen, it will typically have a corresponding private variable of the same name prefixed with an “F”.  In this case you should create a private section and place a variable named FScreen in this section.  I don’t know where the “F” prefix came from, possibly Frank Borland, but like Microsoft’s use of “C” to prefix everything, Borland has chosen an “F”. 

 

The next thing to do is to add the TNScreenScraper unit to the uses clause and add a class variable of TScreenScraper named SS (for screen scraper) in the private section.  Also add a variable named ScreenCompleted of type Boolean in this section as well. 

 

We will also add a TNotifyEvent procedure in the private section as well which will intercept the OnEOS event of the Screen Scraper object.  At this point make sure your code matches the code listed below:

 

unit Unit1;

 

{$WARN SYMBOL_PLATFORM OFF}

 

interface

 

uses

  ComObj, ActiveX, AspTlb, Project1_TLB, StdVcl, TNScreenScraper;

 

type

  TMyASPPage = class(TASPObject, IMyASPPage)

  private

    FScreen: String;  //Private manipulation of Screen property

    SS: TScreenScraper; //our runtime instantiation of Screen Scraper

    ScreenCompleted: boolean; //lets us know when Screen has returned!

 

    procedure MyEOS(Sender: TObject); //this is a TNotifyEvent procedure

 

  protected

    procedure OnEndPage; safecall;

    procedure OnStartPage(const AScriptingContext: IUnknown); safecall;

    function Get_Screen: WideString; safecall;

    procedure ConnectMe; safecall;

    procedure Set_Screen(const Value: WideString); safecall;

  end;

 

implementation

 

uses ComServ;

 

procedure TMyASPPage.OnEndPage;

begin

  inherited OnEndPage;

end;

 

procedure TMyASPPage.OnStartPage(const AScriptingContext: IUnknown);

begin

  inherited OnStartPage(AScriptingContext);

end;

 

function TMyASPPage.Get_Screen: WideString;

begin

   result := FScreen;

end;

 

procedure TMyASPPage.ConnectMe;

begin

     SS := TScreenScraper.Create(nil); //no parent

     SS.Host := 'nerdc.ufl.edu';

     SS.Port := 23;

     SS.OnEOS := MyEOS; //catch the EOS event at runtime rather than at d.t.

 

     ScreenCompleted := false;

     SS.Connect;

 

     repeat

       SS.Breath; //aka Application.ProcessMessages!!

     until (ScreenCompleted);

 

     FScreen := SS.getScreen; //get the raw characters!

 

end;

 

procedure TMyASPPage.Set_Screen(const Value: WideString);

begin

   FScreen := Value;

end;

 

procedure TMyASPPage.MyEOS(Sender: TObject);

begin

      //Catch the SS OnEOS event

      ScreenCompleted := true;

end;

 

initialization

  TAutoObjectFactory.Create(ComServer, TMyASPPage, Class_MyASPPage,

    ciMultiInstance, tmApartment);

end.

 

There are a couple things to point out.  First, we have created at run-time (on-the-fly) a variable named SS which surprisingly has a parent of nil and is accurate.  The next noticeable item is the run-time assignment of the OnEOS event of the SS object.  This is an easy way to capture events in code when you don’t have a form to place objects on and then double click on the events in the object inspector.  We then follow our original plan to continuously loop until ScreenCompleted is assigned true in the OnEOS event MyEOS.  The next item is the SS.getScreen which takes the characters in the screen_buf[] public variable of SS and converts them to standard ascii values and then takes the whole screen as a string.  At this point save your work and save the Project as ScreenScraperSample.

 

The next step is to create or modify the ASP page that will instantiate this object:

 

<HTML>

<BODY>

<TITLE> Testing Delphi ASP </TITLE>

<CENTER>

<H3> You should see the results of your Delphi Active Server method below </H3>

</CENTER>

<HR>

<% Set DelphiASPObj = Server.CreateObject("ScreenScraperSample.MyASPPage")

   DelphiASPObj.ConnectMe

%>

<%=DelphiASPObj.Screen%>

 

<HR>

</BODY>

</HTML>

 

The above sample ASP was created by Delphi when we created the Active Server Object; however, I did modify the script just a bit to now call the ConnectMe() function and then display the Screen property after the emulator connects. 

 

At this point compile the application and make sure your ASP script is exactly as written above.  After compiling and checking the application, register the server by selecting Run | Register ActiveX Server from the menu bar.

 

It is many times necessary to access an ASP page from an active IP server, which means you can’t merely double click on the ASP page and expect it to work.  You should create a Virtual Directory pointing to the directory that houses the ASP page and start it from there! 

 

From Internet Explorer type in the location to the ASP page (i.e. http://127.0.0.1/MyASP/MyASPPage.asp) and press enter.  This will fire off the ASP page which will connect with nerdc.ufl.edu and return back with the text of the screen:

 

 

Pretty vanilla, but very efficient.  The next thing to do is add some more functionality to our COM application.  You will have to shut down your IIS server and restart it in order to make modifications to the ASP COM object.  Make the following changes to your code:

 

procedure TMyASPPage.ConnectMe;

var

k: Word;

begin

     SS := TScreenScraper.Create(nil); //no parent

     SS.Host := 'nerdc.ufl.edu';

     SS.Port := 23;

     SS.OnEOS := MyEOS; //catch the EOS event at runtime rather than at d.t.

 

     ScreenCompleted := false;

     SS.Connect;

 

     repeat

       SS.Breath; //aka Application.ProcessMessages!!

     until (ScreenCompleted);

 

     ScreenCompleted := false;

 

     k := 112;  //Press F1 aka VK_F1

     SS.KeyDown(k,[]);

 

     //Go to Help Screen

     repeat

       SS.Breath;

     until (ScreenCompleted);

 

     ScreenCompleted := false;

     k := 114; //could use VK_F1, VK_F2, VK_F3, etc.

     SS.KeyDown(k,[]);

 

     //exit the Help Screen

     repeat

       SS.Breath;

     until (ScreenCompleted);

 

 

 

     FScreen := SS.GetScreen; //get the raw characters!

 

end;

 

 

 

You will notice that this script will get the first screen, press F1, and then press F3 returning the original screen as when first connected.  Use ASCII key codes when passing in commands to KeyDown and use KeyPress to pass in characters.  We are going to add some more code to travel through the emulation looking for a book on Star Wars.  You might want to first follow along in a regular emulator by going to nerdc.ufl.edu.

 

1)      At the login screen type LUIS

2)      On the next screen type 6

3)      On this screen type T=STAR WARS

 

If all has gone well you should see at least 35 titles on Star Wars.  Navigating and selecting the appropriate books requires that you go through the String in GetScreen() looking for titles that you are interested in.  The following represents the code to accomplish a search on Star Wars:

 

procedure TMyASPPage.ConnectMe;

var

k: Word;

ch: char;

begin

     SS := TScreenScraper.Create(nil); //no parent

     SS.Host := 'nerdc.ufl.edu';

     SS.Port := 23;

     SS.OnEOS := MyEOS; //catch the EOS event at runtime rather than at d.t.

 

     ScreenCompleted := false;

     SS.Connect;

 

     repeat

       SS.Breath; //aka Application.ProcessMessages!!

     until (ScreenCompleted);

 

     ScreenCompleted := false;

 

     k := 112;  //Press F1 aka VK_F1

     SS.KeyDown(k,[]);

 

     //Go to Help Screen

     repeat

       SS.Breath;

     until (ScreenCompleted);

 

 

     ScreenCompleted := false;

     k := 114;

     SS.KeyDown(k,[]);

 

     //exit the Help Screen by Pressing F3!!

     repeat

       SS.Breath;

     until (ScreenCompleted);