Use Tokenizer TAPoRware Tool in Your Web Page

You can add a button in your web page to tokenize this page (in words) by call TAPoRware cgi script.

Here is the code for the tool interface

<form method="post" name="htmlForm" enctype="multipart/form-data" target="_blank" action="http://taporware.mcmaster.ca/~taporware/cgi-bin/prototype/htokenize.cgi" onsubmit="document.htmlForm.htmlurl.value=document.location.href">

<input type="hidden" name="source" value="url" />

<input type="hidden" name="htmlurl" />

<input type="hidden" name="freetext" value="yes"/>

<input type="hidden" name="tagtext" value="body" />

<input type="hidden" name="token" value="Word" />

<input type="hidden" name="dispop" value="1" />

<input type="hidden" name="HowToList" value="1" />

<input type="hidden" name="taporface" value="same" />

<input type="submit" name="doit" value="Tokenize This Tags" />

</form>

• Enter a valid html tag or tag list separated by comma, default is "body"

• Select how you want the separators to be treated

• • htmlTag -- any html element (tag) name or multiple html element name separated by comma

• Endpoint URL: http://taporware.mcmaster.ca:9982
• Service URI: http://taporware.mcmaster.ca/~taporware/webservice

Predefined Parameter Values in Tool Bar

• Source: the page the user is currently in.
• Element: body or set by site owner
• Token type: words
• Display option: strip separator
• Display format: HTML

Term Definition

• token: The basic syntactic unit of a text. A token consists of one or more characters, excluding the blank character and excluding characters within a string constant or delimited identifier.
• tokenizer: A parsing program that scans text and determines when and if a series of characters can be recognized as a token.

• Enter a valid URL in the URL field or enter a local upload html text
• Enter a valid html tag or tag list seperated by comma, default is "body"
• Select token type. If you select "Separate on characters" or "Separate on pattern", you need to provide the corresponding characters or pattern -- in unix style or regular expression
• Select how you want the separators tb be treated
• Select output format
• If you want the results displayed in the same window with taporware interface, uncheck the check box - "Open results in new window"
• Finally, click the "Submit" button

• URL: http://taporware.mcmaster.ca/cgi-bin/prototype/htokenize.cgi

Here are the parameters:

• Endpoint URL: http://strange.mcmaster.ca:9982
• Service URI: http://strange.mcmaster.ca/~taporware/webservice
• Service Method: tokenizer_HTML
• parameters:
  • htmlInput -- any html string
  • htmlTag -- any html element (tag) name or multple html element name separated by comma
  • tokenType -- values can be Word/Line/Sentence/Paragraph/char/pat
  • option1 -- the value of this parameter depend on the "tokenType" you used. If "char" is used, you should enter space separated characters. if "pat" is used, you should enter "unix" or "regexp"
  • option2 -- if you enter "unix" in the option1 field, enter a unix styled pattern, if you enter "regexp" in the option1 field, enter a valid regular expression, otherwise, keep it empty.
  • displayOption -- value can be 1/2/3/4 which is corresponding to strip separator/keep separator as token/keep with previous token/keep with following token
  • outFormat -- value can be 1/2/3 which is corresponding to HTML/XML text in HTML/XML tree respectively

• Obtain HTML string by URL or from user's local disk
• Obtain text contained by user specified tags
• Generate token or separator based on user selected token type
• Perform tokenizer function
• Arrange token and separator based on user's preference
• Generate output format

Tokenize HTML Document

List HTML Tags

Description

History

Pseudocode

Ways of Using

CGI Interface

Web Service Interface

Known Bugs

To Do

-- MattPatey - 13 Oct 2005