## proc processdata

Welcome Gallery Handbook

Manual page for proc_processdata(PL)

## DESCRIPTION

proc processdata may be used to perform various types processing on a data set after it is read.

## FEATURES

Various types of processing may be done, including break processing, accumulation, tabulation and counting, rewriting as percents, computation of totals, reversing record order, and rotation of row/column matrix.`` Only one of these types of processing may be done per invocation of proc processdata, and the type of processing is controlled by the `action` attribute.`` In addition to any one of the actions, data may be further processed in that fields may be filtered (kept or rejected).`` The result may replace the original data set in memory or occupy a new space.`` In either case, after proc processdata is finished the result will be considered the "current data set" which all plotting procs will use.`` The original data set may be returned to later using proc originaldata.``

## EXAMPLE

A Gallery example where this is used is stock where the available data is in reverse chronological order.`` Since lineplot must work from left to right, proc processdata is used to reverse the record order.``

## VARIABLES THAT ARE SET

NRECORDS = Number of rows in the data result.``

NFIELDS = Number of fields per row in the data result.``

TOTALS = If totals, percents, or accumulation are being done, this variable will be set to hold the field total(s). If more than one field is being operated on, this will be a comma-delimited list of totals; individual totals may be accessed in your script using something like the following, which would access the first total in the list: `#set T = $nmember(1,@TOTALS)`

BREAKFIELD1 .. n = Current contents of break fields, when `action` is breaks.``

## MANDATORY ATTRIBUTES

The `action` attribute must be specified, unless just keeping/rejecting fields.``

## ATTRIBUTES

action a

  The type of processing to perform.`` Often the `fields` attribute is used to indicate which field(s) are involved.`` Legal values for a include:

  accumulate Rewrite field as a cumulative series (accumulation).`` The field(s) to operate on must be given in the `fields` attribute.`` For example, the data set on the left would be transformed to the one on the right (`fields: 2`):

      	A21 3			A21 3
      	A22 5		-->	A22 8
      	A23 2			A23 10
      	A24 1			A24 11

  breaks (new in version 1.39). Break processing. In data processing terminology, "break processing" is the act of passing through a sorted data set and taking some action when a key field or fields change.`` Break processing is significantly more efficient than scanning a data set multiple times with a select statement.`` The data set must be sorted such that key fields are grouped.`` The key field(s) must be specified using the `fields` attribute.`` The BREAKFIELD1 .. n variable(s) will be set to the current contents of the break field(s). Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).`` A gallery example that uses this feature is mouse See MORE ON BREAK PROCESSING below for more information.``

  breakreset Reset the "current row" to the beginning of the data set, for the occasional time when more than one pass through a data set will be done using the breaks action.``

  count Collapse data by counting the number of instances of a key field. Input data must be sorted (or at least grouped) on the key field.`` Resulting data set will always have two fields.`` One or two fields must be specified using the `fields` attribute. If one field is specified, the result fields will be 1) key field, 2) count.`` For example (`fields: 1`):

      	062698	 		062698 2
      	062698 		  -->	062898 1
      	062898 			070198 3
      	070198 			070498 1
      	070198
      	070198
      	070498

  If two fields are specified, the result fields will be 1) key field, 2) sum of the numeric contents found in the second specified field.`` For example (`fields: 1 2`):

      	062698 4		062698 10
      	062698 6	  -->	062898 3
      	062898 3		070198 9
      	070198 2		070498 2
      	070198 4
      	070198 3
      	070498 2

  See also the gallery example hitcount (Introduced in version 1.3)

  percent Rewrite one or more fields as percentages of its field (column) total.`` The field(s) to operate on must be given in the `fields` attribute.`` For example (`fields: 1`):

      	8		40
      	4	   -->	20
      	3		15
      	5		25

  reverse The last record becomes the first one; the record order is reversed. For example (`fields: 2`):

      	AXB 34		DIF 14
      	BYA 22	   -->	CES 52
      	CES 52		BYA 22
      	DIF 14		AXB 34

  rotate First row becomes 1st field, 2nd row becomes 2nd field, and so on. This may be useful in that most of the plotting procs work from data fields, but sometimes data is given (or is more intuitive) in rows.`` For example:

      	A 2 4 6 8 10	-->	A B
      	B 3 6 9 12 15		2 3
      				4 6
      				6 9
      				8 12
      				10 15

  select Select certain records.`` Result will be made up of records meeting the condition given in the `select` attribute.``

  total Compute field total(s) only and place total(s) into the variable TOTALS (see above). This action does not rewrite data.`` The field(s) to operate on must be given in the `fields` attribute.``
  The decimal format of the total(s) is controlled by the `resultformat` attribute. If total(s) are to be written in presentable notation (a spacer for thousands, etc.) the `resultformat` attribute may be preceded by a `n`, e.g. `n%7.0f`.``

fields dfield list

  The field(s) to be operated on when doing percents, accumulations, or totals.``
  Example: `fields: 2 5 6 7`

keepfields dfield list

  If specified, only the listed fields in the original data set will be kept. The others will be rejected. `action` may be anything.``
  Example: `keepfields: 4 5 6`

rejectfields dfield list

  If specified, the listed fields in the original data set will not be kept. The others will be. `action` may be anything.``
  Example: `rejectfields: 1 2 3 4`

fieldnames namelist

  If specified, the names given in namelist may be used in any plotting proc to identify data fields. namelist is a space- or comma- delimited list of names. Names may include any alphanumeric characters with a maximum length of 38, and are case-insensitive.``
  Note that if field names are specified in proc getdata and then proc processdata is used to alter the order of fields or delete fields, then this `fieldnames` attribute must be used in order to redefine the field names properly.``
  Example: `fieldnames: date group n`

resultformat printf-spec

  Controls the decimal format of rewritten percents, accumulations, totals.``

keepall `yes` | `no`

  If `yes`, original fields are preserved when doing accumulate, percent, or total.`` Thus the result will have more fields than the original data set. Default is `no`.``

select selection-expression

  Used when `action` is `select` in order to specify the selection condition.

stack `yes` | `no`

  Normally the results of the processing take the place of the original data set in memory. However, if `stack` is set to `yes`, the results become the current data set and the original data set is preserved (it may be restored as the 'current' data set by invoking proc originaldata).`` This may be useful when working with larger data sets, to avoid re-reading them from disk.``

showresults `yes` | `no`

  If `yes` the data are shown after processing, as a diagnostic aid.``

## MORE ON BREAK PROCESSING

The breaks action (new in version 1.39) allows break processing. In data processing terminology, "break processing" is the act of passing through a sorted data set and taking some action when a key field or fields change.`` For example, if we were processing a list of charges ordered by paying budget number, we could use a break processing strategy to pause and generate a statement for one budget number, when we reached the point in the data set where the budget numbers changed. Then we would continue on.`` The breaks action allows a similar thing to be done with plotting.`` Break processing is significantly more efficient than scanning the entire data set multiple times with a select statement, especially with larger data sets.``

The data set must be sorted such that key fields are grouped.`` The key field(s) must be specified in the `fields` attribute.`` proc processdata is generally called within a #loop. When proc processdata finishes, the "current data set" will be the block of data from the previous break to the current break. Subsequent invocations of proc processdata will continue from the previous location in the data set. The original data set is always assumed (i.e. it is not necessary to use proc originaldata).

Your script can access the current contents of the break field(s) via the BREAKFIELD1 .. n variable(s).`` Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).`` The following is an example:

     	#loop
     	   #proc processdata
     	     action: breaks
     	     fields: 1 2 3
     	   #proc endproc

     	   #if @NRECORDS = 0
     	     #break
     	   #endif

     	   #proc page
     	   title: Account @BREAKFIELD1

     	   #proc bars
     	   ...

     	 #endloop

Limits: up to 5 break fields may be used.`` Comparisons for equality are limited to the first 50 characters.``
data display engine
Copyright Steve Grubb

Markup created by unroff 1.0, `` `` June 18, 2001.