#set TITLE = "proc getdata"
#include top
 
.SH DESCRIPTION
\fBproc getdata\fR is used to access or define data for plotting.
\fBproc getdata\fR must be executed before any
data-based plotting can be done.

.SH FEATURES
Data may be specified literally within the script file, 
or it may be located in an external file or 
be the result of an externally executed
command.  
.LP
Data may be specified at the end of the script file using
#intrailer/ #proc trailer (see EXAMPLES, below).
.LP
Conditional selection of data records.
.LP
Data filtering for purposes such as combining or rearranging fields,
performing calculations, truncating outliers, etc.



.SH EXAMPLES
Virtually all of the Gallery examples use #proc getdata.


.SH ACCEPTABLE DATA FORMATS
See
.ig >>
<a href="dataformats.html">
.>>
dataformats
.ig >>
</a>
.>>


.SH PREREQUISITES
None.


.SH VARIABLES THAT ARE SET
The variable NRECORDS will be set to the number of records read,
and the variable NFIELDS will be set to the number of fields per record.


.SH ERROR CONDITIONS
If no data records were read, NRECORDS will be set to zero.
This may be tested using script code such as this, which generates
an image containing the words "No data found" :
.IP
.nf
\0#proc getdata
\0   ...
\0
\0#proc endproc
\0#if @NRECORDS = 0
\0  #proc annotate
\0  location: 3 3
\0  text: No data found.
\0
\0  #exit
\0#endif
.fi
.LP


.SH SETTING VARIABLES FROM A DATA FILE
Data files may contain embedded \fB#set\fR commands to set ploticus variables.
This may be a convenient way for title, labels, etc. to be driven by the data file.
The syntax is the same as in ploticus scripts: 
.IP
\fC#set \fIvarname\fC = \fIvalue\fR
.LP
\fIvarname\fR will be declared and set to \fIvalue\fR.  \fIvalue\fR may
contain embedded spaces or any other character.  \fIvalue\fR may not
be a $function(), nor may it contain any other @variables.
None of the other script operators, such as \fB#if\fR, are supported in this context.



.SH SETTING DATA FIELD NAMES 
If you wish, you may assign names to data fields, and then be able to reference
data fields by name.  If names are not assigned,
fields must be referenced by field number, where the first field is 1.
Use of field names may allow more readable code.
Field names may be assigned in the ploticus script file using the \fCfieldnames\fR
attribute.  Field names may also be embedded in the data file (see the \fCfieldnameheader\fR
attribute).


.SH MANDATORY ATTRIBUTES
Either \fCdata\fR, \fCfile\fR, or \fCcommand\fR must be specified
(see below).


.SH ATTRIBUTES
.LP
\fBdata\fR 
.ig >>
<a href="attributetypes.html#text">
.>>
\fI multiline-text \fR
.ig >>
</a>
.>>
.IP
Literal specification of plotting data.  Terminates at
first blank (empty) line.  Example:
.nf
.ft C
data:   "Case 1"   0   4   4.1   
        "Case 2"   1   5   4.4  
        "Case 3"   2   2   4.0 
        "Case 4"   3   9   4.8

.fi
.ft R
.LP
\fBfile\fR \fIfilename\fR
.IP
Shell-expandable name of a file containing plotting data.
This name will be used along with cat(1) in a shell command, thus
exported shell variables and metacharacters may be part of the name.
A dash (-) may be used if data is to be read from the standard input,
(or the \fCstandardinput\fR attribute may be used).
Example: \fCfilename: myfile.dat\fR

.LP
\fBpathname\fR \fIfilename\fR
.IP
Name of a file containing plotting data.
The file will be opened directly.
This may be preferable for emulation environments where basic
unix commands such as cat(1) are not available.
Shell variables and metacharacters may not be used.

.LP
\fBcommand\fR \fIshell command line\fR
.IP
A shell command that will produce plot data on its standard output.
.br
Example: \fCcommand: cat mydat | uniq -c\fR
 
.LP
\fBfieldnames\fR \fInamelist\fR
.IP
If specified, the names given in \fInamelist\fR may be
used in any plotting proc to identify data fields.  \fInamelist\fR is a 
space- or comma- delimited list of names.  Very lengthy lists 
(> 200 characters long) should be space-delimited.  Names may include 
any alphanumeric characters with a maximum length of 38, and are case-insensitive.
Embedded spaces are not allowed.
.br
Example: \fCfieldnames: date group n\fR

.LP
\fBfieldnameheader\fR  \fCyes\fR | \fCno\fR
.IP
If \fCyes\fR, the first non-comment line in the data is expected to
hold a list of field names.  See the \fCfieldnames\fR attribute for more information
about field names.
If the \fCdata\fR attribute is used, use of \fCfieldnames\fR
is preferred over \fCfieldnameheader\fR.

.LP
\fBstandardinput\fR \fCyes\fR | \fCno\fR
.IP
If \fCyes\fR, data is read from the standard input.

.LP
\fB#intrailer\fR
.IP
Indicates that a \fCdata\fR attribute
will be given in a #proc trailer, at the end of the script file.
See EXAMPLES, below.

.LP
\fBcommentchar\fR 
.ig >>
<a href="attributetypes.html#string">
.>>
\fI string \fR
.ig >>
</a>
.>>
.IP
A character or group of characters that is
used to signify a comment in the data file.
Commented lines will be skipped.
Default is \fC//\fR.
Example: \fCcommentchar: #\fR

 
.LP
\fBdelim\fR \fCspace\fR | \fCcomma\fR | \fCtab\fR
.IP
The type of delimiting method to be used when 
parsing the data.  See
.ig >>
<a href="dataformat.html">
.>>
dataformat
.ig >>
</a>
.>>
for details.
.br
Example: \fCdelim: comma\fR

.LP
\fBshowresults\fR \fCyes\fR | \fCno\fR
.IP
If \fCyes\fR, the results, after selecting and/or filtering, are
written to the diagnostic file,
which may be useful in debugging, etc.


.LP
\fBrotate\fR  \fCyes\fR | \fCno\fR
.IP
Allows data to be given all in one row, even when plotting proc
expects one record per instance (which most do).
Only applicable if your data set has one row.
(There still must be a blank line following the data attribute.)
To rotate more than one row, use \fBproc processdata\fR.
.br
Example: see 
.ig >>
<a href="../gallery/bars1.htm">
.>>
bars1 
.ig >>
</a>
.>>

.LP
\fBselect\fR 
.ig >>
<a href="condex.html">
.>>
\fI conditional-expression \fR
.ig >>
</a>
.>>
.IP
This allows data records to be selected for inclusion based
upon a selection expression.  
Incoming data fields are referenced by number,
using a double at-sign (\fC@@@@\fR) prefix.  
\fCselect\fR cannot be used along with the \fCdata\fR attribute.
Hint: use the \fCshowresults\fR attribute when debugging.
.br
Example: \fCselect: @@@@3 = g\fR
.br
This would select all data records having 3rd field equal to \fCg\fR.

.LP
\fBfilter\fR 
.ig >>
<a href="attributetypes.html#text">
.>>
\fI multiline-text \fR
.ig >>
</a>
.>>
.IP
An embedded script which processes incoming data records.
Typical uses are for:
concatenating or splitting fields,
doing on-the-fly date conversions,
or generating derived fields such as the
sum of several fields or the difference between two fields.
.br
.br
.IP
The embedded script will be applied once to every incoming data record.
The script should produce some "output"; generally the last statement is a
\fB##print\fR.  
The output must use the same delimitation method as the input.
The script uses the same syntax
as the greater ploticus script, except that:
.RS
.IP \(bu
directives must begin with two pound signs (\fC##\fR) instead of one
.IP \(bu
local variables begin with two at signs (\fC@@@@\fR) instead of one
.IP \(bu
fields on the incoming data record are accessed like this: \fC@@@@1\fR for
the first field, \fC@@@@2\fR for the second, etc.  If you are using field
names, these may be used as well, eg: \fC@@@@score\fR.
.IP \(bu
the only directives that may be used are 
\fB##set\fR, \fB##if\fR, \fB##elseif\fR, \fB##else\fR, \fB##print\fR, \fB##call\fR,
and \fB##exit\fR
.RE
.IP
Other things worth noting:
.RS
.IP \(bu
the filter script is terminated using a blank line.
.IP \(bu
use the \fCshowresults\fR attribute when debugging.
.IP \(bu
if \fCfilter\fR is used along with \fBselect\fR, the \fCselect\fR is applied first.  
.IP \(bu 
\fCfilter\fR cannot be used along with the \fCdata\fR attribute.
.IP \(bu
\fCfilter\fR can only access fields from one data record at a time.
.IP \(bu
variables from the ploticus script may be referenced within
the \fCfilter\fR script (use one at-sign \fC@@\fR).
Evaluation occurs before the filter script executes.
.IP \(bu
ploticus script #if/#else statements (single pound sign)
may be used to selectively execute portions of the filter script.
Interpretation occurs before the filter script executes.
.IP
Example: This filters out data records
having field 2 or field 3 equal to M.  It then
calulates the difference in days between two dates
and puts this difference in the variable DIFF.
Finally it "prints" incoming field 1 along with DIFF.
Thus the result of this #proc getdata
will have be data records having two fields.
.nf
.ft C
\0filter:
\0     ##declare DIFF
\0     ##if @@@@2 = M || @@@@3 = M
\0       ##exit
\0     ##endif
\0     ##set DIFF = $daysdiff(@@@@3,@@@@2)
\0     ##print @@@@1 @@@@DIFF
.fi
.ft R
.IP
There are several more \fCfilter\fR examples in the 
.ig >>
<a href="../doc/Faq.html#filter">
.>>
FAQ
.ig >>
</a>
.>>


.SH MORE EXAMPLES
.LP
Data specification may be located at the end of the script file
by using \fB#intrailer\fR and \fB#proc trailer\fR.  This may be
useful in "getting the data out of the way", or with automated building
of script files.
Here is how this is done:
.nf
.ft C
\0#proc getdata
\0#intrailer

\fIother #procs, etc.\fR

\0#proc trailer
\0Data:	0.3 0.5 2.3
	3.5 9.4 1.4
	\fI..etc..\fR
\fIend of file\fR


#include bottom
