Dataset Transformation Map

Overview

A dataset transformation map is a transformation which calculates a value for each record in the dataset, but the transformation requires data from the entire dataset (or at least a large set of records from the dataset).

There are multiple methods to accomplish this type of map, some are demonstrated below. Example 1
The first example calculates the dataset aggregate wihtin the map function. The function passed to map takes three arguments, the current item, the current index, and the full dataset.

In this example, a 3 period moving average is calculated by taking the last 3 elements of the data argument for each item and calculating an average.

This method of calculating is slower than other methods, because it recalculates the average from scratch for each item in the list, without re-using any part of the calculating between items.

let data = [{price:100},{price:101},{price:100},{price:102},{price:102},{price:103},{price:105},{price:102}];

let result = data.map((p,i,data)=>{
  let item = {
    price:p.price,
    ma:null
  };
  if(i>=2) item.ma = $list(data).last(3).map(p=>p.price).average();
  return item;
})
				

Example 2
The second example calculates the moving averages as an array of numbers, and then appends those numbers onto each record through the map mehtod. This method is generally quicker because the calculation of the moving average array can make use of prior calculations.

let ma = await import('/lib/time-series/moving-average/v1.0.0/moving-average.mjs');

let data = [{price:100},{price:101},{price:100},{price:102},{price:102},{price:103},{price:105},{price:102}];
let maData = ma.movingAverage(data.map(p=>p.price),3);

let result = data.map((p,i)=>{
  return {
    price:p.price,
    ma:maData[i]
  }
})
				

Try it!

Example 3
The merge method of list functions in a like manner to the example above, in that it takes a calculated array of values and appends them to the objects in the list.

let ma = await import('/lib/time-series/moving-average/v1.0.0/moving-average.mjs');

let data = [{price:100},{price:101},{price:100},{price:102},{price:102},{price:103},{price:105},{price:102}];

let result = $list(data).merge(data=>ma.movingAverage(data.map(p=>p.price),3), 'ma');
				

Try it!