The isp definitions are written in javascript and structured with as a javascript object. Version 3 of isps.js is designed for greater flexibility in the procedure which data is obtained from the ISP website and offers more customisation of ISP-specific options. The API allows for:
The isps.js file is located inside the content folder of the netusageitem.jar extension package. Extract the xpi installation package to find the jar package. Both packages are a Zip archive and can be opened using 7-Zip or WinRar.
Version 2.1.5 allows you to develop an ISP by going to the preferences and selecting the debug tab.
The basic procedure for net usage to fetch usage information invloves executing a series of steps, which could include running a javascript function, downloading a web page via GET or POST and parsing out the variables from the last retrieved web page using either xpath (for XML pages) or regular expressions for scraping normal HTML pages.
Very often, a storage is required to share information between steps. This is done via an associative array (called vars) which can store arbitary variables. It is also used by store the final download/upload/usage values obtained from the fetching process, which the meter then uses to display on the GUI.
A basic ISP definition will look like this:
{
id: 'unwired'.
group: 'Australia',
name: 'Unwired',
process: [
'POST https://www.unwired.com.au/myaccount/login.php?ev=nx login_name={USERNAME}&login_password={PASSWORD}&loginbutton=Login',
[/Login failed/, "ERROR", /./],
'USAGEPAGE',
'GET https://www.unwired.com.au/myaccount/usage.php',
/My current plan is: ([A-Za-z0-9 \/]+)[^`]+?My current usage allowance:[\s]+([0-9A-Za-z]+)[^`]+?usage.php">([A-Za-z0-9\-]+) - ([A-Za-z0-9\-]+)[^`]+?Total<\/th>[\r\n ]+<th[^>]*>([0-9\.]+)<\/th>[\r\n ]+<th[^>]*>([0-9\.]+)<\/th>[\r\n ]+<th[^>]*>([0-9\.]+)/,
'GET https://www.unwired.com.au/myaccount/logout.php',
],
varsDisplay: {
'1,0': {
'NONE': '{1:1|planname} {1:3|startdate} {1:4|enddate}',
'Usage': '{1:7|MB|usage} / {1:2|AB|usage-quota}',
'Upload': '{1:5|MB|upload}',
'Download': '{1:6|MB|download}',
}
},
}
The isps.js contains an array variable named nu_isps. The array of isps specifies the isp id as the key and a isp definition structure as the value. The isp id is a unique identifier for the isp. The isp id has a recomended convention of countrycode.ispname (eg. nz.telstraclear). Because of historical reasons, Australian isps are not prefixed.
The isp definition structure is an associative array containing the following keys:
idAn id for the ISP - should be alpha-numeric with no spaces
groupThe group that the ISP should appear under. For australian ISPs, the group should be Australia
name The full name of the ISP
description (optional) This is the text displayed for ISP specific preferences in the preference dialog. Normally corresponds to preference options, ie. "Please enter your download allowance"
vars (optional) This is an associative array containing the initial values for vars array. Special vars include basicrealm for basic authentication.
prefs (optional) This is an associative array of variables that the user can enter through netusage's preference dialog.
process This is the most important value in the entire definition. It is an array that describes a series of steps to perform in order to obtain the required usage values. The step number starts with 0 for the first step and increments to the next step after the successful completion of each step. However the next step can be arbitarily specified according to the result of a parsing step. In addtion, there are control flow commands such as SKIPNEXTSTEP, STARTUSAGEPAGE and EXIT. See below for the type of steps you can use.
varsDisplay This structure has two purposes: a mapping for variables and the display for the GUI tool tip display. It is linked with parsing as different parsing branches can have different variable mapping and tool tip display. The MAPVARS command is implicitly run at the end of the process, unless explicitly specified inside the process array. Depending on the step and branch number taken during the process, different tooltips can be displayed.
Three different types of steps can be specified: Commands, Parsing, Functions. Most isps only need commands and parsing steps. The process part of the ISP definition typically looks like this:
process: [
'POST http://www.somearbitraryfancyisp.com/usage/login.cgi username={USERNAME}&password={PASSWORD}',
[
/Invalid username or password/, 'ERROR',
/./
],
'GET http://www.somearbitraryfancyisp.com/usage/usage.cgi',
/Usage: ([0-9]+)MB.+?Limit: ([0-9]+)MB/,
'GET http://www.somearbitraryfancyisp.com/usage/logout.cgi'
]
There are many commands with the most important ones being POST and GET.
Commands are written as a javascript string with capatilised command followed by parameters, all separated by a space. Variables can be embedded in the string, by specifying curly braces and the name of the variable. These variables are stored in the vars array used throughout the fetching process.
eg. 'POST http://www.somearbitraryfancyisp.com/usage/login.cgi username={USERNAME}&password={PASSWORD}'
The variables are automatically escaped with encodeURIComponent() as it is mostly used in the context of HTTP POST and GET query strings. If you want a raw string, add the NOESC modifier to the variable {MYVAR|NOESC}. For more details on modifiers, please see variable mapping.
There are POSTBASIC and GETBASIC commands for authenticating in pages protected by Basic Authentication. If you are using Basic Auth, please fill also add a basicrealm variable to the vars array (i.e. with the vars key) if you want to visit ISP page function to authenticate automatically as Firefox's password manager session requires a matching realm name.
After a page is fetched with GET or POST commands, they are stored in a buffer ready to be parsed.
For a full list of commands, see below.
The parsing step can be a single regular expression or xpath mapping or an array of both. If it is an array then individual regular expression and xpath mappings are called branches. These branches are useful because a single URL web page for an ISP may be slightly different for different users that have different plans (ie ones with Off peak quota vs ones that don't). In an array the parsing is done one by one until it one of the branches matches.
If parsing of this branch is successful (matches), the next step in the process is usually performed. However if in the parsing array, immediately following the parsing branch is a string or an integer element, then the next step will be the step in the integer or the label in the string. These strings and integers are in the same level as the branches are ignored during matching.
[ /Invalid username or password/, 'ERROR', /./ ],
In the above example, two branches can be seen in the parsing. The first regexp branch matches the string "Invalid username or password" and the second regexp branch matches anything. This is a typical parsing step used to verify that the username and password was accepted by the ISP's usage facility. If the resultant page after the login step contained the text "Invalid username or password", then it meant that the login has failed. The step to jump to if that happens, is "ERROR", which is a special jump point to the end that will yeild a Invalid username message in the netusage gui front end. If the text "Invalid username or password" does not match, then the next branch is considered, which in this case, matches anything (special regexp modifier dot). There is no string or integer element after the regexp, which means, it'll jump to the next step by default.
If no parsing match has been found in the array, then a parsing error will be raised. To make it fall through, so the process continues even if no matches are found, a default regular expression /./ should be added as the last item of the parsing array (like the above example).
Any captures from the regular expression will be put into the vars array, with a key name of parse number, followed by colon and capture number. The first set of parsing branches, will be 0, and the first capture inside it will be 1, thus having a vars key name of 0:1. To re-use a capture on a subsequent command, you can use {0:1} in the params of the command.
/Usage: ([0-9]+)MB.+?Limit: ([0-9]+)MB/,
In the above regular expression, the second parsing, the resultant match of first bracketted sub pattern is put into the the vars array with a key of 1:1, the match for the second bracket, 1:2.
TODO explain xpath
Arbitary functions can be performed in the ISP array. The return value of the function is the next step that should be performed (jumped to). It can be an integer for a step number or a label. Note that using functions should be a last resort and you should try to use commands and parsing to achieve all the functionality required.
Example of a function that jumps to the label SUPERPLAN if plan_type variable is super.
function (vars) {
if (vars['plan_type'] == 'super')
return 'SUPERPLAN',
},
The varsDisplay structure has dual purposes. One to reference for post process variable mapping, and second to display a custom tooltip tailored for this particular ISP.
Since the display of the tooltip can vary due to the parsing, the varsDisplay holds a set of tooltips and in most cases, only one of the will take effect.
The keys of the associative array is a string specify the parse number and branch which the process ran through. If the first parsing matched the third regexp pattern (or xpath), then the structure with the key '0,3' will apply.
varsDisplay: {
'0,1': {
},
'0,2': {
},
'0,3': {
}
}
Variable Mapping basically copies variables in vars array into different keys, applying any specified modifiers. This happens automatically at the end of the normal fetching process. You can also explicitly call this action in the process with the MAPVARS command, in which case force the mapping earlier so you can apply post-process functions on mapped variable in the process array.
Variable Mapping is important because there are predefined variables in the vars array which affect the final GUI output. These variables are:
download
download-quota
usage
usage-quota
downloadpo
downloadpo-quota
usagepo
usagepo-quota
usagepc (usage percentage)
download2
download2-quota
usage2
usage2-quota
usagepc2 (usage2 percentage)
The download/usage and quota conterparts are what affects the main usage gauge display. Use the usage var if the ISP counts both uploads and downloads. The usage variable takes precedence over the download var, so if the usage var is detected present, it will be used for the GUI display; if not, the download variable will be used. If present, the downloadpo/usagepo, download2/usage2 variables affect the second usage gauge display. The precidence rules mirror the download/usage vars. Use the *po variables if the second metering is time sensitive (ie, an offpeak meter). Otherwise, if its accounting is active 24/7 then you should use download2 (ie, if the second meter is international traffic).
Mapping of a variable is written as elements separated by pipes enclosed in parenthesis. The first element is the source variable name (key) and the last element is the variable name (key) to copy the result to. Anything else in between are the modifiers. Modifiers apply a function to modify the variable, like converting a MB usage number to the standard B usage number. Modifiers are applied left to right. There are built in modifiers (AB, GB, MB, KB, B, COMMADEC, NOESC).
You can also use variables and modifiers in commands, also enclosed in parenthesis and elements separated by pipes. These have no destination variable name element as they are not for mapping. A useful modifier is NOESC which forces the variable replacement step not to URI Encode the rendered variable.
There numerous built-in vars that you can use in your functions or commands.
lasttext - contains the response text (eg html) of the last downloaded URL
lasturl - contains the url of the last request (corresponds with lasttext)
USERNAME - the username specified by the user in preferences
PASSWORD - the password specified by the user in preferences
now_month - current month, ie, 1 for January
now_month_3l - current month but in 3 letter form eg, Jan, Feb
now_day - current day of month
now_year - current year eg 2008
ispid - isp id as specified by the developer in this file
ispname - isp name as specified by the developer in this file
TODO
These commands and case sensitive. Parameters, if any, are in italics and specified after the command.
LABEL labelname - Not really a command, but a named jump destination.
USAGEPAGE - Not really a command, but a special label that specifies the usage page for the Visit ISP Usage Page option in the menu. If the user is visiting the ISP usage page and the process reaches this page, this page is shown to the user and the process is terminated.
STARTUSAGEPAGE - Not really a command, but a special label signifying that after this point, the steps are for the Visit ISP Usage Page option. If normal fetching/parsing reaches this label, the process is terminated (and MAPVARS runs). If the user is visiting the ISP usage page, the process starts on the STARTUSAGEPAGE step.
REFRESHREDIRECT - catches if a page has a http-equiv redirect that sends the browser to another page. This command loops until there is no more redirects and then proceeds to the next step.
GET url
GETBASIC url
POST url querystring
POSTBASIC url querystring - These commands fetches a URL with differing methods.
See Commands for details.
MAPVARS - A command explicitly run the variable mapping. Commonly used if a post processing step is required to manipulate mapped variables. See Variable mapping and Tooltip display for details.
EXIT - Terminates the process (and run MAPVARS if it hasn't run)
SKIPNEXTSTEP - Skips the next step. If current step is step 2 (SKIPNEXTSTEP), next will be step 4.