File Data Processor

Overview


The file data processor library reads files and then applies mapping and filtering to the data as it is read into memory. This allows you to do mapping and filtering on files that are too large to be in memory at one time. The library can be found here: file-data-processor.mjs

Mapping and Filtering




(async function(){
  const fd = await import('./file-data.mjs');
  let data = await fd.lines('c:/path/data.txt')
                      .filter(p=> p !== '')
                      .map(p=> p.substr(3));
})()


Read and parse csv files. 


(async function(){
  const fd = await import('./file-data.mjs');
  let data = await fd.csv('c:/path/data.csv')
                      .filter(p=> p.age > 10')
                      .map(p=> p.name);
})();


Read and parse json files 


(async function(){
  const fd = await import('./file-data.mjs');
  let data = await fd.json('c:/path/data.json')
                      .filter(p=> p.age > 10')
                      .map(p=> p.name);
})();


In the case that you want to read a large file and do something with the results as they pass by, but not keep anyting in memory, you do your processing in a filter method that always returns false. 


(async function(){
  const fd = await import('./file-data-processor.js');
  await fd.csv('c:/path/data.csv')
           .filter(p=>{
		      //..do something					
		      return false;
	       });            
})();

Contents