Overview
Javascript has an effective limit of 500mb for the size of a string. That is, text that exceeds 500mb in size cannot be stored within a single variable. This limit does not affect arrays, so for instance, text that exceeds the limit can be handled by storing each line as a separate string within an array.
However, data streamed from a web server is streamed as text. This means, that any data that comes from a web server must either be less than 500 mb, or be served as multiple files.
Retrieving Multiple Files
When a folder on the server is queried, the davinci file server will return the names of the files within that folder. That means that one can just retrieve the list of files, and then query each file in turn for its data, adding the results to a single array.
let urls = await $url("http://localhost:2800/folder1/");
let data = [];
for(let url of urls){
let data1 = await $url("http://localhost:2800/folder1/"+url);
data1.forEach(p=>data.push(p));
}
Server API
The server library encapsulates the logic above into a single function.
let sv = await import('/lib/server/v1.0.0/server.mjs');
let server = sv.server($url);
let urls = await server("http://localhost:2800/folder1/");