Creating Screen Scrapers with Visual 3270 for ActiveX
First, what is a screen scraping application or Host Rejuvenation application? In its simplest form it is the simple process of “scraping” data from a traditional “green screen” or terminal that communicates with a mainframe or host system:
In the above screen shot, the OpenConnect Emulator has connected to the University of Florida and has a continuous TCP/IP link to the system. Host Rejuvenation programs will translate this “raw” data from its vanilla form above and convert it into another form, usually HTML. Even more importantly is the fact there is no install on the host device and as such represents an attractive solution for PC developers using Visual C++, Visual Basic, Delphi, Java, and C++Builder. Such applications can include:
College Course Registration On-Line
Vehicle Registration On-Line
Parking Tickets On-Line
Single Sign-on Applications
As you can see, this list is only a small portion of a large market and can include Government applications, Medical, Commercial, and Education. For instance, OpenConnect’s Visual 3270 for ISAPI converts “green screens” into pure HTML versions as shown below:
The above application was written entirely in Object Pascal using Delphi 6.0 and is the definitive thin client application. The above application can support up to 500 simultaneous users without significant resources on the server that is processing the requests. Undoubtedly you may be wondering about the architecture behind this system:
Obviously the most important component is the COM object in the Active Server Page which will communicate with the out of process server or in-process server.
Building your ASP Object
Visual Basic is very easy to create Active Server Pages and makes programming a TN3270 Emulator much easier than other programming languages. The first thing to do is to open up Visual Basic and select File | New Project from the menu bar. From this dialog select the ActiveX DLL project and press the Open button similar to the following screen shot.
In the Project Explorer window you will likely see a simple project named appropriately Project1 and a class named obviously class1. You will notice that there are no forms to work with since this is a DLL and even if a Form is provided will not be visible at runtime. The next thing to do is to add a reference to this ActiveX DLL so that Visual Basic understands that this will be used as an Active Server Page rather than just some DLL. To do this select Project | References from the menu bar. This should bring up a dialog box which will display all the available Type Libraries registered with the system. If you don’t yet know what a type library is don’t worry, you’re going to make one in just a minute. For those that need to know now, a type library is an interface definition that allows differing processes to communicate with objects without knowing the underlying logic, all that is necessary is to “know” the rules for communication—thus type libraries were invented.
If you haven’t already done so, from the References dialog box look for the Microsoft Active Server Object line item:
As can be seen from the above dialog box, there can be several Active Server Pages Object Libraries, but notice that this one points to asp.dll which is what we want. As long as the reference points to asp.dll you’re in business. Press Ok to complete the process of adding an ASP object to our ActiveX DLL project.
At this point let’s save the project and the class to a name that can be easily remembered, like MyProjectASP and MyClassASP. To do this select the File | Save Project As menu item and save your project and class with the suggested names. Your Project Explorer should now look like the following:
You may have noticed that you didn’t actually assign a name to Project1 and Class1, which is to be expected since all we did was save the files with the suggested file names. You should now add some names that you like by right mouse clicking on Project1 in the Project Explorer, selecting Properties and giving it a name:
Do the same for the Class1:
If you got lost on that, just try and get your Project Explorer to look like the above images as this is what we will be using when we reference our object from an ASP page.
The next step is to add a special routine and code so that our web page can send and retrieve information from the ASP object. If you’re not very comfortable with programming ASP objects in Visual Basic there are literally hundreds of books on the subject and we suggest you play with developing ASP pages without Objects just to get the hang of utilizing server side and client side scripting in ASP.
For the rest of us we must now add the code to communicate with the ASP HTML-like web page. In the code view of the MyASPClass type the following code:
Option Explicit
Private
m_ScriptingContext As ScriptingContext
Public Sub
OnStartPage(ByVal theScriptingContext As ScriptingContext)
Set m_ScriptingContext =
theScriptingContext 'make the assignment!
End Sub
The OnStartPage subroutine is called by Internet Information Server when an ASP web page requests the services of the object. In this subroutine we are capturing the ScriptingContext object and saving it in our private variable m_ScriptinContext for later updating and usage. The ScriptingContext object provided by IIS packs a lot of functionality and we suggest you learn more about this object. For now just know that we have to have it in order to return and retrieve information from the client web page.
In order to successfully use the TNEmulatorX control you must add it as a REFERENCE to your project. Add the OCX reference to the project similar to when we added the ASP object. Since the TNEmulatorX object is an ActiveX control it does not usually appear in the standard view of the References dialog box and as such you must search for it and add the TNEmulatorXControl1.OCX to the project. This object is usually found in the install directory but is not the default extension so remember to change the view of the extensions to OCX:
Once you have added this reference the line to create the object will now function correctly as demonstrated in the code to follow. A Screen Scraper does not incorporate a visual graphics image for keyboard entry and other terminal like emulation, it merely manages a TN3270 session and its associated protocol; however, since its an ActiveX control it still has the same amount of weight as a visual emulator. Developer’s should consider using the COM object for lightweight computing.
ADDING A CONNECT ROUTINE
Its time to add the routine that will be called from ASP. This function will not return until the Scraper has completed its session with the mainframe at which time any data can be displayed in the HTML portion. Make sure your code now looks exactly like the following:
Option Explicit
Private
m_ScriptingContext As ScriptingContext
Private
WithEvents TNScraper As TNEmulatorXControl1.TNEmulatorX
Public Sub
OnStartPage(ByVal theScriptingContext As ScriptingContext)
Set m_ScriptingContext =
theScriptingContext 'make the assignment!
End Sub
Public Sub
Connect()
Set TNScraper =
CreateObject("TNEmulatorXControl1.TNEmulatorX")
End Sub
Ok, we’re finally getting somewhere. In the above Connect() routine the most important aspect of the code is the:
Set TNScraper = CreateObject(“TNEmulatorXControl1.TNEmulatorX”).
This is the most important function call in the whole program because it creates our Screen Scraper object at run time for use during this session. You can now use the Screen Scraper as any other control. Also notice the use of WithEvents prior to declaring the TNScraper private variable, this will allow you to add the necessary events to the code and have them fire during execution.
I would like to take a moment and discuss the fundamental difference between OpenConnect’s ActiveX control and other controls—Asynchronous Events. The OpenConnect controls do not use threading techniques in windows and as such, events can happen at any given time, including end of screens, end of records, receipt of data, and other TCP/IP related events. Just remember it for now, it will become clearer as we move along.
The next thing to do is to add the code to connect to the University of Florida. We are going to create a session to U.F., go to its library and information services section, look for a title on Star Wars and return the information back to the web page.
The first thing to do is to add the properties to connect to the host. The mandatory properties are Host, and Port with other properties such as TN3270E, LUName, etc. For more information on these properties please review the help file, otherwise make sure your code looks like the following:
Option
Explicit
Private
m_ScriptingContext As ScriptingContext
Private
WithEvents TNScraper As TNEmulatorXControl1.TNEmulatorX
Private
mEOS As Boolean
Private
myResponse As Response
Public
Sub OnStartPage(ByVal theScriptingContext As ScriptingContext)
Set m_ScriptingContext =
theScriptingContext 'make the assignment!
End
Sub
Public
Sub Connect()
Dim S As String
Set TNScraper =
CreateObject("TNEmulatorXControl1.TNEmulatorX")
'This is so we can send text back to the
web page!
Set myResponse =
m_ScriptingContext.Response
'Set up the connection
TNScraper.Host =
"nerdc.ufl.edu"
TNScraper.Port = 23
'This gets set to true when first screen
returns
mEOS = False
TNScraper.Connect
'this method works efficiently but
doesn't
'check the screen contents. Developers
'shoudl consider using InStr() on the
'screen contents before moving forward.
Do
DoEvents
Loop Until (mEOS)
'At this point we should be at the first
'screen of the University of Florida's
'mainframe. We want to type LUIS
'at the command prompt
'Lets send the screen back to the web
page
S = TNScraper.getScreen
myResponse.Write S
End
Sub
Public
Sub TNScraper_OnEOS()
'This event is fired when a mainframe has
'sent a completed screen!
mEOS = True
End Sub
In the above code you may have noticed the TNScraper_OnEOS() event that we placed manually into the code. It works in conjunction with the Do..Loop in the Connect() event. Remember when we talked about Asynchronous events? This is an example of an asynchronous event in which we don’t know when the screen will return so we have to use the mEOS private variable to let us know when to break out of the Do…Loop. After the screen returns, the OnEOS event fired, we simply grab all the text using the GetScreen() function and then send the text back to the web page by using the myReponse variable. Developers should also note that many mainframes send multiple OnEOS events which is typical when moving data arrives similar to a web page. You should make sure that the screen has in fact returned by either knowing for sure that there is one OnEOS event by utilizing the VBEmulator “Visual” control looking for OnEOS events, or check the screen text by using the InStr() function.
SENDING KEYSTROKES TO THE SESSION.
There are two types of keystrokes when using the VBEmulator and match directly with windows events KeyDown and KeyPress. KeyDown events and functions are used to send character code type data and include:
ENTER
F1
F2
F3
F4
.
.
F12
If you have used KeyDown or KeyPress then using them in the emulator is same as if using them in a regular VB application. KeyPress is strictly character data and includes the standard alphanumeric characters and other “keyboard” characters. If you are not familiar with the differences review the Visual Basic documentation, or just look at the following code and you should get the hang of it. Basically, when you want to enter characters, use KeyPress, when you want to send Attention Identifiers (AID) use KeyDown. Make your connect() sub routine now looks like the following:
Public
Sub Connect()
Dim S As String
Set TNScraper =
CreateObject("TNEmulatorXControl1.TNEmulatorX")
'This is so we can send text back to the
web page!
Set myResponse =
m_ScriptingContext.Response
'Set up the connection
TNScraper.Host =
"nerdc.ufl.edu"
TNScraper.Port = 23
'This gets set to true when first screen
returns
mEOS = False
TNScraper.Connect
'this method works efficiently but
doesn't
'check the screen contents. Developers
'shoudl consider using InStr() on the
'screen contents before moving forward.
Do
DoEvents
Loop Until (mEOS)
'At this point we should be at the first
'screen of the University of Florida's
'mainframe. We want to type LUIS
'at the command prompt
'Testing the screen!
'Lets send the screen back to the web
page
'S = TNScraper.GetScreen
'myResponse.Write S
'Our cursor should be positioned where we
want it
'if not you can use CursorLocation(x) to
set
'the cursor in the screen buffer.
The Screen
'buffer is usually made up of a Model 2
screen
'comprising 1920 consecutive characters
with
'each 80th character starting a new row
(24x80).
TNScraper.KeyPress (Asc("L"))
TNScraper.KeyPress (Asc("U"))
TNScraper.KeyPress (Asc("I"))
TNScraper.KeyPress (Asc("S"))
mEOS = False
'Lets send an Enter command
'NOTICE HOW IT CONFORMS TO STANDARD
KEYDOWN!
TNScraper.KeyDown 13, 0
Do
DoEvents
Loop Until (mEOS)
'Testing the screen!
'Lets send the screen back to the web
page
S = TNScraper.GetScreen
myResponse.Write S
End Sub
When building an ASP page you should check your ASP page after each screen to make sure you are in deed getting the right data. Clients will generally add a WriteLog function to see what is taking place and is highly suggested. Clients will also add a Timeout algorithm by adding a Form to the project and drop a Timer object on it turning it on prior to a Do..Loop and either turning it off when an EOS (End of Screen) or timing out the function and immediately returning a “timeout” web page to the user.
If you have followed along successfully and you have run the application using the ASP script below:
<HTML>
<BODY>
<%
set mySock =
Server.CreateObject("MyASPProject.MyASPClass")
mySock.Connect
%>
</BODY>
</HTML>
If you have done everything correctly, including adding a reference to this page as a Virtual Directory on your IIS Server, you should have received the following screen:
Its ugly, but once you get the data you can do whatever you want with it. The next section will demonstrate how to enter a long string of text without having to continually call KeyPress over and over again for each character and will complete our search for a book on Star Wars:
Public
Sub Connect()
Dim S As String
Set TNScraper =
CreateObject("TNEmulatorXControl1.TNEmulatorX")
'This is so we can send text back to the
web page!
Set myResponse =
m_ScriptingContext.Response
'Set up the connection
TNScraper.Host =
"nerdc.ufl.edu"
TNScraper.Port = 23
'This gets set to true when first screen
returns
mEOS = False
TNScraper.Connect
'this method works efficiently but
doesn't
'check the screen contents. Developers
'shoudl consider using InStr() on the
'screen contents before moving forward.
Do
DoEvents
Loop Until (mEOS)
'At this point we should be at the first
'screen of the University of Florida's
'mainframe. We want to type LUIS
'at the command prompt
'Testing the screen!
'Lets send the screen back to the web
page
'S = TNScraper.GetScreen
'myResponse.Write S
'Our cursor should be positioned where we
want it
'if not you can use CursorLocation(x) to
set
'the cursor in the screen buffer. The Screen
'buffer is usually made up of a Model 2
screen
'comprising 1920 consecutive characters
with
'each 80th character starting a new row
(24x80).
TNScraper.KeyPress (Asc("L"))
TNScraper.KeyPress (Asc("U"))
TNScraper.KeyPress (Asc("I"))
TNScraper.KeyPress (Asc("S"))
mEOS = False
'Lets send an Enter command
'NOTICE HOW IT CONFORMS TO STANDARD
KEYDOWN!
TNScraper.KeyDown 13, 0
Do
DoEvents
Loop Until (mEOS)
'Testing the screen!
'Lets send the screen back to the web
page
'S = TNScraper.GetScreen
'myResponse.Write S
'Lets enter a "6" to go to
FSU's library!!
TNScraper.KeyPress (Asc("6"))
mEOS = False
TNScraper.KeyDown 13, 0
Do
DoEvents
Loop Until (mEOS)
'Good time to test the screen
'S = TNScraper.GetScreen
'myResponse.Write S
'Exit Sub
'Lets look for a book on Star Wars!!!
TNScraper.EnterString "T=STAR
WARS", True
mEOS = False
TNScraper.KeyDown 13, 0
'This is a good loop but it doesn't give
us any
'way to exit out. Should add a timer to
'this!!!
Do
DoEvents
S = TNScraper.GetScreen
If InStr(1, S, "Search
Results", vbTextCompare) > 0 Then
Exit Do
End If
Loop Until (False)
myResponse.Write S
End
Sub
You may notice immediately two new techniques, first is the EnterString() function which will place a string of text into the emulator all at once. The second is the use of the IntStr() to exit out of the Do..Loop when we receive the results we’re looking for. If you have followed along correctly and have accessed the ASP page created earlier, then your browser should look like the following:
You will notice there are 35 entries for this selection. My favorite is the Timothy Zahn series. You can look through this string looking at positions to get the number of entries (35) and each entry. You can send any character to the emulator by using the KeyPress and KeyDown methods but remember to try them out in the Emulator first before using them in a non-visual version of your application.
If you need more assistance, please call OpenConnect today and we will try to answer your questions. Also check out the FAQ’s at our website!