Generators
Overview
Sample Generator Code
def count_up_to(max_num):
count = 1
while count <= max_num:
yield count # Pauses execution and returns the current number
count += 1 # Resumes here on the next call
Chaining Together Data Sources
Generators makes it easy to comvine multiple data sources into a what appears as a single data source.
The following function takes two file paths, and merges them into a single data source using the
generator mechanism.
import json
def stream_json_files(file1_path, file2_path):
# Combine the file paths into a list to loop through them
files = [file1_path, file2_path]
for file_path in files:
# Open and load one file at a time
with open(file_path, 'r', encoding='utf-8') as f:
data_list = json.load(f)
# Ensure the root element is actually a list
if isinstance(data_list, list):
for item in data_list:
yield item
else:
raise ValueError(f"File {file_path} does not contain a JSON list.")
Analytics on Generators
Once a data source is created using a generator, there are some challenges to building analytics
using the resulting generator.
Consider the case where we wish to calculate two statistics, the average and max of sequence of numbers given
by a generator. You have the following functions
def average(data):
sum = 0
count = 0
for item in data:
count+=1
sum+=item
pass
return sum/count
def max(data):
val=None
for item in data:
if val==None:val=item
elif item>val:val = item
return val
You could in theory calculate these as follows:
data = get_generator()
av = average(data)
mx = max(data)
But there are problems with this code. First, a generator can only be used once. THat is, you can only iterate
trough a generator one time, so only the average will be computed. You could create a separate generator for
each calculation, but then you are still iterating through the data twice, and if the data is large, this is
wasteful and time consuming.
The following code iterates through the generator only once, and runs both calculations
sum=0
count=0
max=None
for item in data:
count += 1
sum += item
if max=None:max=item
elif item>max:max = item
pass
average = sum/count
However now we have intermingled the logic of each calculation. We can no longer define our calculation as
an independent function that encapsulates the logic.