Balancing
Exploration and Exploitation for Efficient Black-Box Cloning in Smart
Manufacturing
(Code
and results of experiments related to the article submitted to ISM-2025:
“International Conference on Industry of the Future and Smart Manufacturing”)
ABSTRACT
The
increasing complexity of smart manufacturing systems demands advanced digital
twinning solutions for process optimization. A key challenge is the cloning of
hidden functions, where direct access to complex systems or human experts is
restricted or expensive. This paper introduces an adversarially
guided cloning framework that leverages hybrid learning strategies—balancing
exploitation (efficient weight optimization) and exploration (intelligent input
selection)—to replicate black-box functions under constrained query budgets and
temporal drift. We propose an adversarial sampling strategy, inspired by active
learning, adversarial training, and curriculum learning, to select informative
queries and improve cloning efficiency. Through controlled experiments, we
demonstrate that our method outperforms random sampling in replicating hidden
supervisory functions, particularly in scenarios with limited access to ground
truth. Additionally, we investigate cloning under temporal drift conditions,
where the supervisor’s decision boundaries evolve over time, requiring adaptive
strategies to maintain cloning accuracy. While our study focuses on synthetic
experiments, future research will explore cloning multiple interrelated
supervisor functions and integrating them into a unified decision model for
complex industrial processes. Our approach contributes to AI-driven digital
twin enhancements by enabling more efficient modeling of unknown industrial
processes with minimal data while preserving reasonable precision.
CODE AND EXPERIMENTS
Part I (Cloning Hidden Supervisor)
1.1. Hidden supervisor as a binary classifier
=========================================
# BEGINNING OF THE CODE
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
# ------------------------ Hyperparameters ------------------------
eta_x = 0.01 # Step size for
gradient-based updates on input points
k_values = list(range(1, 11, 1)) # Different values of k controlling the
frequency of random vs. gradient-based samples
num_runs = 10 # Number of
independent runs for each k value
N_desired = 500 # Total number of training points to be generated per
run
# Student NN Configuration
num_hidden_layers = 2 # Number of hidden layers in the student network
hidden_layer_size = 20 # Number of neurons in each hidden layer
# ----------------------------------------------------------------
# Define the student (clone) model
class SimpleNN(nn.Module):
def __init__(self, num_hidden_layers, hidden_layer_size):
super(SimpleNN, self).__init__()
self.layers = nn.ModuleList()
self.layers.append(nn.Linear(3, hidden_layer_size)) # Input layer with
3 input features
for _ in range(num_hidden_layers - 1):
self.layers.append(nn.Linear(hidden_layer_size, hidden_layer_size)) # Hidden layers
self.layers.append(nn.Linear(hidden_layer_size, 20)) # Intermediate
layer before output
self.layers.append(nn.Linear(20, 1)) # Output layer with a single neuron
self.sigmoid = nn.Sigmoid() # Sigmoid
activation function for output
def forward(self, x):
for i in range(len(self.layers) - 1):
x = torch.relu(self.layers[i](x)) # Apply ReLU activation to hidden layers
x = self.sigmoid(self.layers[-1](x)) # Apply Sigmoid
activation to final output
return x
# Define the supervisor (complex analytical function) for classification
# This function maps input vectors to binary labels using a nonlinear
transformation
def supervisor_function(x):
output = torch.sin(x[:, 0]) + torch.cos(x[:, 1]) * torch.sigmoid(x[:, 2] / 2)
return (output > 0).int() # Convert to binary classification: 1 if output
> 0, otherwise 0
# Function to calculate accuracy of the model on an evaluation dataset
def calculate_accuracy(model, eval_data_loader):
model.eval() # Set model to
evaluation mode (disables dropout, etc.)
correct = 0
total = 0
with torch.no_grad(): # Disable gradient computation for efficiency
for data, target in eval_data_loader:
outputs = model(data) # Forward pass
predicted = (outputs > 0.5).int() # Convert probabilities to binary predictions
total += target.size(0) # Track total
number of samples
correct += (predicted.squeeze(1) == target).sum().item() # Count correct predictions
return 100 * correct / total # Compute percentage accuracy
# Generate evaluation data (fixed dataset for testing model performance)
num_eval_samples = 1000 # Number of evaluation samples
X_eval = torch.randn(num_eval_samples, 3) # Randomly
generate input features
y_eval = supervisor_function(X_eval) # Generate labels using the supervisor function
eval_dataset = torch.utils.data.TensorDataset(X_eval, y_eval) # Create dataset
object
eval_data_loader = torch.utils.data.DataLoader(eval_dataset, batch_size=32, shuffle=False) # Data loader for
batch processing
# Store results and generated training points
results = {} # Dictionary to
store average accuracy for each k
input_points = {} # Dictionary to
store the generated points for each k
# Loop over different values of k
for k in k_values:
accuracies = [] # List to store accuracy values for multiple runs
input_points[k] = [] # List to store
generated points per k
for run in range(num_runs): # Repeat
experiment for multiple runs
model = SimpleNN(num_hidden_layers, hidden_layer_size) # Instantiate the
student model
optimizer = optim.SGD(model.parameters(), lr=0.01) # Stochastic Gradient Descent optimizer
train_losses = [] # List to track loss
values
current_run_points = [] # Store training
points generated in the current run
generated_points_count
= 0 # Counter to track
total generated points
# Define a single loss function for training
criterion = nn.BCEWithLogitsLoss() # Binary Cross
Entropy with Logits Loss
while generated_points_count < N_desired:
# Continue until the required number of points is reached
if generated_points_count % k == 0: # Generate a
random point every k iterations
x = torch.randn(1, 3) # Randomly sample
a new input point
x.requires_grad_(True) # Enable gradient
tracking for x
current_run_points.append((x.detach().numpy(), 'blue')) # Mark as randomly generated
# Perform model
training using the randomly generated point
optimizer.zero_grad() # Reset gradients
supervisor_output
= supervisor_function(x) # Get label from supervisor function
model_output
= model(x) # Forward pass
through the model
loss = criterion(model_output, supervisor_output.float().unsqueeze(1)) # Compute loss
loss.backward() # Compute
gradients
optimizer.step() # Update model
parameters
train_losses.append(loss.item())
# Store loss value
else:
x = x.detach().clone().requires_grad_(True) # Ensure x is a
fresh tensor with gradient tracking
with torch.no_grad():
# Compute the supervisor output without gradient computation
supervisor_output = supervisor_function(x)
model_output
= model(x) # Forward pass
loss_gradient_ascent
= criterion(model_output, supervisor_output.float().unsqueeze(1)) # Compute loss
loss_gradient_ascent.backward() # Compute
gradients w.r.t. x
if x.grad is not None:
x = x
+ eta_x * x.grad.detach()
# Update x using gradient ascent step
current_run_points.append((x.detach().numpy(), 'red')) # Mark as gradient-based update
optimizer.zero_grad() # Reset gradients
before the next update
model_output
= model(x) # Forward pass
again
loss = criterion(model_output, supervisor_output.float().unsqueeze(1)) # Compute loss
loss.backward() # Compute
gradients
optimizer.step() # Update model
parameters
train_losses.append(loss.item())
# Store loss value
generated_points_count
+= 1 # Increment point
counter
input_points[k].append(current_run_points)
# Store the training points for this run
# Evaluate model performance after training
accuracy = calculate_accuracy(model, eval_data_loader) # Compute accuracy
accuracies.append(accuracy)
# Store accuracy for this run
print(f'Run: {run + 1}, k: {k}, Accuracy: {accuracy:.2f}%') # Display results
results[k] = np.mean(accuracies)
# Compute average accuracy over all runs
print(f'k: {k}, Average
Accuracy: {results[k]:.2f}%') # Display final average accuracy
# END OF THE CODE
=========================================
Summary of
experiments:
Hidden
supervisor:
|
# -------------- Hyperparameters (COMMON FOR ALL THE EXPERIMENTS BELOW) ------------------ eta_x = 0.01 # Input update
step for the gradient-based update with regard to inputs k_values = list(range(1, 11, 1)) # Values for k
(after each k iterations a random sample is generated) num_runs = 10 # Number of runs for each k (used for sufficient statistics
to get reliable measurements) N_desired = CHANGES OVER EXPERIMENTS # Total number of generated training points
(key parameter) #
---------------------------------------------------------------------------------------------------------------------------------- # Hidden “Supervisor”: complex analytical function aka decision boundary
for binary classification: # def supervisor_function(x): output = torch.sin(x[:,
0]) + torch.cos(x[:, 1]) * torch.sigmoid(x[:,
2] / 2) return (output
> 0).int() # Return 1 if output > 0,
else 0 # use the function to
label samples #
---------------------------------------------------------------------------------------------------------------------------------- # “Student” NN configuration (“Student” NN will learn to classify
inputs imitating the “Supervisor”) num_hidden_layers = 2 # Number of hidden layers hidden_layer_size = 20 # Number of neurons in each hidden
layer # ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
EXPERIMENT # 1: N_desired = 20000 |
EXPERIMENT # 2: N_desired = 10000 |
|
69 min (» 7 min per run) |
29 min (» 3 min per run) |
|
k:
1, Average
Accuracy: 98.67% k: 2, Average Accuracy: 98.21% k: 3, Average Accuracy: 97.86% k: 4, Average Accuracy: 97.99% k: 5, Average Accuracy: 97.35% k: 6, Average Accuracy: 97.63% k: 7, Average Accuracy: 97.59% k: 8, Average Accuracy: 97.25% k: 9, Average Accuracy: 97.45% k: 10, Average Accuracy:
96.68% … k: 20000, Average Accuracy: 59.52% |
k: 1, Average Accuracy: 96.53% k: 2, Average Accuracy: 95.52% k: 3, Average
Accuracy: 96.76% k: 4, Average Accuracy: 95.95% k: 5, Average Accuracy: 96.13% k: 6, Average Accuracy: 95.49% k: 7, Average Accuracy: 95.61% k: 8, Average Accuracy: 94.65% k: 9, Average Accuracy: 94.62% k: 10, Average
Accuracy: 94.61% … k: 10000, Average Accuracy: 53.08% |
|
EXPERIMENT # 3: N_desired = 5000 |
EXPERIMENT # 4: N_desired = 1000 |
|
15 min (» 1.5 min per
run) |
3 min (» 18 sec per run) |
|
k: 1, Average Accuracy: 91.76% k: 2, Average Accuracy: 87.19% k: 3, Average
Accuracy: 92.65% k: 4, Average Accuracy: 91.27% k: 5, Average Accuracy: 90.66% k: 6, Average Accuracy: 91.69% k: 7, Average Accuracy: 89.02% k: 8, Average Accuracy: 90.95% k: 9, Average Accuracy: 88.08% k: 10, Average Accuracy: 89.01% … k: 5000, Average Accuracy: 56.36% |
k: 1, Average Accuracy: 55.80% k: 2,
Average Accuracy: 59.14% k: 3, Average Accuracy: 57.31% k: 4, Average Accuracy: 56.54% k: 5, Average Accuracy: 54.92% k: 6, Average Accuracy: 58.57% k: 7, Average Accuracy: 55.67% k: 8, Average Accuracy: 56.16% k: 9, Average Accuracy: 58.43% k: 10, Average Accuracy: 56.41% … k: 1000, Average Accuracy: 55.28% |
|
EXPERIMENT # 5: N_desired = 500 |
EXPERIMENT # 6: N_desired = 200 |
|
1 min (» 6 sec per run) |
55 sec (» 5.5 sec per
run) |
|
k: 1, Average Accuracy: 53.30% k: 2, Average Accuracy: 51.95% k: 3, Average Accuracy: 52.67% k: 4, Average
Accuracy: 55.33% k: 5, Average Accuracy: 52.68% k: 6, Average Accuracy: 51.86% k: 7, Average Accuracy: 55.19% k: 8, Average Accuracy: 52.48% k: 9, Average Accuracy: 51.03% k: 10, Average Accuracy: 51.81% … k: 500, Average Accuracy: 55.88% |
k: 1, Average Accuracy: 53.12% k: 2, Average Accuracy: 51.13% k: 3, Average Accuracy: 51.13% k: 4, Average Accuracy: 50.28% k: 5, Average Accuracy: 51.60% k: 6, Average Accuracy: 51.60% k: 7,
Average Accuracy: 53.33% k: 8, Average Accuracy: 51.93% k: 9, Average Accuracy: 51.54% k: 10, Average Accuracy: 52.00% … k: 200, Average Accuracy: 50.00% |
|
EXPERIMENT # 7 (extreme): N_desired = 100000 |
EXPERIMENT # 8 (extreme): N_desired = 100 |
|
k: 1,
Average Accuracy: 98.66% … k >
1 accuracy steadily declines … k: 100000, Average Accuracy: 51.58% |
k: 1, Average Accuracy: 50.40% k: 2, Average Accuracy: 49.92% k: 3, Average Accuracy: 51.72% k: 4,
Average Accuracy: 55.09% k: 5, Average Accuracy: 52.71% k: 6, Average Accuracy: 54.01% k: 7, Average Accuracy: 51.18% k: 8, Average Accuracy: 50.17% k: 9, Average Accuracy: 50.04% k: 10, Average Accuracy: 53.68% … k: 100, Average Accuracy: 50.04% |
1.2. Hidden supervisor as a regression model
=========================================
# BEGINNING OF THE CODE
!pip install torch torchvision
import torch
import torch.nn as nn
import torch.optim as optim
import numpy
as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d
import Axes3D
%matplotlib inline
# Define the student (clone) model
class SimpleNN(nn.Module):
def __init__(self):
super(SimpleNN, self).__init__()
self.layer1 = nn.Linear(3, 10)
self.layer2 = nn.Linear(10, 1)
def forward(self, x):
x = torch.relu(self.layer1(x))
x = self.layer2(x)
return x
# Define the supervisor (complex analytical function)
with sigmoid
def supervisor_function(x):
return torch.sin(x[:, 0]) + torch.cos(x[:, 1]) * torch.sigmoid(x[:, 2] / 2)
# Hyperparameters
eta_x = 0.01
num_epochs = 100 # Reduced for faster execution
k_values = list(range(1, 101, 5)) # Reduced for faster execution
num_runs = 5 # Reduced for faster execution
# Store results and input points
results = {}
input_points = {}
# Loop over k values
for k in k_values:
mse_values =
[]
input_points[k]
= []
for run in range(num_runs):
model = SimpleNN()
optimizer = optim.SGD(model.parameters(),
lr=0.01)
train_losses
= []
for epoch in range(num_epochs):
if epoch % k == 0:
x = torch.randn(1, 3, requires_grad=True)
input_points[k].append((x.detach().numpy(), 'blue'))
else:
with torch.no_grad():
if x.grad is not None:
x.data += eta_x * x.grad
x.grad = None if x.grad is None else x.grad.zero_()
input_points[k].append((x.detach().numpy(), 'red'))
x_normalized = (x - x.mean())
/ x.std()
model_output = model(x_normalized)
supervisor_output = supervisor_function(x_normalized)
loss_fn = nn.MSELoss()
loss = loss_fn(model_output, supervisor_output)
optimizer.zero_grad()
loss.backward(retain_graph=True)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
train_losses.append(loss.item())
if epoch % 10 == 0: # Print loss every 10 epochs
print(f"k = {k}, run = {run}, epoch = {epoch}, loss = {loss.item()}")
x_eval
= torch.randn(100, 3)
clone_outputs
= model(x_eval)
supervisor_outputs
= supervisor_function(x_eval)
mse
= loss_fn(clone_outputs, supervisor_outputs)
mse_values.append(mse.item())
print(f"k = {k}, run = {run}, MSE = {mse.item()}")
results[k] = np.mean(mse_values)
for k, mse
in results.items():
print(f"k = {k:4d}, Average Mean Squared Error
(MSE): {mse}")
# 3D Visualization
x_range = np.linspace(-3, 3, 50)
y_range = np.linspace(-3, 3, 50)
X, Y = np.meshgrid(x_range, y_range)
Z = np.zeros_like(X)
for i in range(X.shape[0]):
for j in range(X.shape[1]):
input_tensor
= torch.tensor([[X[i, j], Y[i, j], 0.0]])
Z[i, j] = supervisor_function(input_tensor).item()
fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis', alpha=0.5)
for k, points in input_points.items():
for point, color in points:
ax.scatter(point[0][0], point[0][1], point[0][2], color=color,
alpha=0.7, s=20)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Supervisor Output')
ax.set_title('Supervisor Function
Projection and Input Points')
plt.show()
# END OF THE CODE
=========================================
Samples of
experiments:
k = 1,
Average Mean Squared Error (MSE): 0.5854145288467407
k = 6,
Average Mean Squared Error (MSE): 0.5637600421905518
k = 11,
Average Mean Squared Error (MSE): 0.6558600068092346
k = 16,
Average Mean Squared Error (MSE): 0.6206687569618226
k = 21,
Average Mean Squared Error (MSE): 0.6220848202705384
k = 26,
Average Mean Squared Error (MSE): 0.6322223901748657
k = 31,
Average Mean Squared Error (MSE): 0.7422250151634217
k = 36,
Average Mean Squared Error (MSE): 0.6371499598026276
k = 41,
Average Mean Squared Error (MSE): 0.7656771540641785
k = 46,
Average Mean Squared Error (MSE): 0.6779741764068603
k = 51,
Average Mean Squared Error (MSE): 0.6234682559967041
k = 56,
Average Mean Squared Error (MSE): 0.7523088097572327
k = 61,
Average Mean Squared Error (MSE): 0.6300775051116944
k = 66,
Average Mean Squared Error (MSE): 0.8710681557655334
k = 71,
Average Mean Squared Error (MSE): 0.877585768699646
k = 76,
Average Mean Squared Error (MSE): 0.7262073397636414
k = 81,
Average Mean Squared Error (MSE): 0.7115568041801452
k = 86,
Average Mean Squared Error (MSE): 0.7018811821937561
k = 91,
Average Mean Squared Error (MSE): 0.8243893504142761
k = 96,
Average Mean Squared Error (MSE): 0.9334649443626404
-------------------------------
k = 1, Mean
Squared Error (MSE): 0.8597227334976196
k = 11, Mean
Squared Error (MSE): 0.8913534879684448
k = 21, Mean
Squared Error (MSE): 0.7325361967086792
k = 31, Mean
Squared Error (MSE): 0.7570369839668274
k = 41, Mean
Squared Error (MSE): 0.5235531330108643
k = 51, Mean
Squared Error (MSE): 0.9116361141204834
k = 61, Mean
Squared Error (MSE): 0.601828396320343
k = 71, Mean
Squared Error (MSE): 1.049517273902893
k = 81, Mean
Squared Error (MSE): 0.7138707041740417
k = 91, Mean
Squared Error (MSE): 0.8350974321365356
k =
101, Mean Squared Error (MSE): 0.9313033223152161
k =
111, Mean Squared Error (MSE): 0.5595588088035583
k =
121, Mean Squared Error (MSE): 1.0405994653701782
k =
131, Mean Squared Error (MSE): 0.6289553642272949
k =
141, Mean Squared Error (MSE): 0.6760601997375488
k =
151, Mean Squared Error (MSE): 0.9233958721160889
k =
161, Mean Squared Error (MSE): 0.7175140380859375
k =
171, Mean Squared Error (MSE): 0.8144766092300415
k =
181, Mean Squared Error (MSE): 0.7933023571968079
k =
191, Mean Squared Error (MSE): 0.635612428188324
k =
201, Mean Squared Error (MSE): 0.5352540612220764
k =
211, Mean Squared Error (MSE): 0.8137766718864441
k =
221, Mean Squared Error (MSE): 0.5526362061500549
k =
231, Mean Squared Error (MSE): 0.5969602465629578
k =
241, Mean Squared Error (MSE): 1.0459617376327515
k =
251, Mean Squared Error (MSE): 0.7793875932693481
k =
261, Mean Squared Error (MSE): 0.5471004247665405
k =
271, Mean Squared Error (MSE): 0.7139944434165955
k =
281, Mean Squared Error (MSE): 0.7137590050697327
k =
291, Mean Squared Error (MSE): 0.6317924857139587
k =
301, Mean Squared Error (MSE): 0.6126051545143127
k =
311, Mean Squared Error (MSE): 0.577974796295166
k =
321, Mean Squared Error (MSE): 0.5321160554885864
k =
331, Mean Squared Error (MSE): 0.6441854238510132
k =
341, Mean Squared Error (MSE): 1.1967272758483887
k =
351, Mean Squared Error (MSE): 0.6476286053657532
k =
361, Mean Squared Error (MSE): 0.7283496856689453
k =
371, Mean Squared Error (MSE): 0.9020192623138428
k =
381, Mean Squared Error (MSE): 0.7173904180526733
k =
391, Mean Squared Error (MSE): 0.503637433052063
k =
401, Mean Squared Error (MSE): 0.8878021240234375
k =
411, Mean Squared Error (MSE): 0.6055228114128113
k =
421, Mean Squared Error (MSE): 0.4629114866256714
k =
431, Mean Squared Error (MSE): 0.5568968057632446
k =
441, Mean Squared Error (MSE): 0.8438667058944702
k =
451, Mean Squared Error (MSE): 0.6329076290130615
k =
461, Mean Squared Error (MSE): 1.3138177394866943
k =
471, Mean Squared Error (MSE): 0.47528016567230225
k =
481, Mean Squared Error (MSE): 0.46840763092041016
k =
491, Mean Squared Error (MSE): 0.909098744392395
k =
501, Mean Squared Error (MSE): 0.5366293787956238
k =
511, Mean Squared Error (MSE): 0.7490972876548767
k =
521, Mean Squared Error (MSE): 0.9888677000999451
k =
531, Mean Squared Error (MSE): 0.6411856412887573
k =
541, Mean Squared Error (MSE): 0.4736836850643158
k =
551, Mean Squared Error (MSE): 0.7271414995193481
k =
561, Mean Squared Error (MSE): 0.5814042687416077
k =
571, Mean Squared Error (MSE): 0.5940609574317932
k =
581, Mean Squared Error (MSE): 0.5540820360183716
k =
591, Mean Squared Error (MSE): 0.7246787548065186
k =
601, Mean Squared Error (MSE): 1.297818899154663
k =
611, Mean Squared Error (MSE): 0.5726361870765686
k =
621, Mean Squared Error (MSE): 0.4536072313785553
k =
631, Mean Squared Error (MSE): 0.6333404183387756
k =
641, Mean Squared Error (MSE): 1.0183906555175781
k =
651, Mean Squared Error (MSE): 0.6050320863723755
k =
661, Mean Squared Error (MSE): 1.1663609743118286
k =
671, Mean Squared Error (MSE): 0.7548099160194397
k =
681, Mean Squared Error (MSE): 1.0954976081848145
k =
691, Mean Squared Error (MSE): 0.8902326226234436
k =
701, Mean Squared Error (MSE): 0.8130860328674316
k =
711, Mean Squared Error (MSE): 0.8108120560646057
k =
721, Mean Squared Error (MSE): 0.7665743231773376
k =
731, Mean Squared Error (MSE): 0.927735447883606
k =
741, Mean Squared Error (MSE): 0.6586292386054993
k =
751, Mean Squared Error (MSE): 0.586413562297821
k =
761, Mean Squared Error (MSE): 0.7356320023536682
k =
771, Mean Squared Error (MSE): 0.8591206073760986
k =
781, Mean Squared Error (MSE): 0.695708692073822
k =
791, Mean Squared Error (MSE): 0.6964506506919861
k =
801, Mean Squared Error (MSE): 1.1372040510177612
k =
811, Mean Squared Error (MSE): 0.7076416015625
k =
821, Mean Squared Error (MSE): 0.8280580043792725
k =
831, Mean Squared Error (MSE): 0.7169986963272095
k =
841, Mean Squared Error (MSE): 0.924439549446106
k =
851, Mean Squared Error (MSE): 0.5267452597618103
k =
861, Mean Squared Error (MSE): 0.43336021900177
k =
871, Mean Squared Error (MSE): 0.610353946685791
k =
881, Mean Squared Error (MSE): 0.8880270719528198
k =
891, Mean Squared Error (MSE): 0.6651954054832458
k =
901, Mean Squared Error (MSE): 0.9224388599395752
k =
911, Mean Squared Error (MSE): 0.6888171434402466
k =
921, Mean Squared Error (MSE): 0.6686494946479797
k =
931, Mean Squared Error (MSE): 1.3357974290847778
k =
941, Mean Squared Error (MSE): 1.0982404947280884
k =
951, Mean Squared Error (MSE): 1.2610973119735718
k =
961, Mean Squared Error (MSE): 0.6645214557647705
k =
971, Mean Squared Error (MSE): 0.8385875821113586
k =
981, Mean Squared Error (MSE): 0.7171139717102051
k =
991, Mean Squared Error (MSE): 0.5657204985618591
Part II (Cloning Hidden Supervisor under Influence of
Temporal Drift)
=========================================
# BEGINNING OF THE CODE
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import math
# Hyperparameters
eta_x = 0.01
k_values = list(range(1, 11, 1))
num_runs = 5
N_desired = 5000
drift_amplitude = 0.5
drift_frequency = 0.001
num_hidden_layers = 2
hidden_layer_size = 20
# Student Model
class SimpleNN(nn.Module):
def __init__(self, num_hidden_layers, hidden_layer_size):
super(SimpleNN, self).__init__()
self.layers = nn.ModuleList()
self.layers.append(nn.Linear(3, hidden_layer_size))
for _ in range(num_hidden_layers - 1):
self.layers.append(nn.Linear(hidden_layer_size, hidden_layer_size))
self.layers.append(nn.Linear(hidden_layer_size, 20))
self.layers.append(nn.Linear(20, 1))
self.sigmoid = nn.Sigmoid()
def forward(self, x):
for i in range(len(self.layers) - 1):
x = torch.relu(self.layers[i](x))
x = self.sigmoid(self.layers[-1](x))
return x
# Supervisor Function with drift based on generated_points_count
def supervisor_function(x, generated_points_count, drift_amplitude, drift_frequency):
drift = drift_amplitude * math.sin(generated_points_count * drift_frequency)
output = torch.sin(x[:, 0]) + torch.cos(x[:, 1]) * torch.sigmoid(x[:, 2] / 2)
output_with_drift = output + drift
return (output_with_drift > 0).int()
# Accuracy Calculation WITH synchronized drift
def calculate_accuracy(model, eval_data_loader, train_points_with_drift, drift_amplitude, drift_frequency):
model.eval()
correct = 0
total = 0
with torch.no_grad():
for i, (data, target) in enumerate(eval_data_loader):
outputs = model(data)
# Calculate drift
for the current batch
batch_drifts
= [drift_amplitude * math.sin((i * eval_data_loader.batch_size +
j + 1) * drift_frequency )
for j in range(data.shape[0])]
drifts_tensor
= torch.tensor(batch_drifts, dtype=torch.float32,
device=outputs.device).unsqueeze(1)
# Apply drift to
the outputs
outputs_with_drift
= outputs + drifts_tensor
predicted = (outputs_with_drift
> 0.5).int()
total += target.size(0)
correct += (predicted.squeeze(1) == target).sum().item()
return 100 * correct / total
# Generate Evaluation Data
num_eval_samples = 1000
X_eval = torch.randn(num_eval_samples, 3)
# generated_points_count for evaluation is 0:
eval_generated_points_count = 0
y_eval = supervisor_function(X_eval,
eval_generated_points_count, drift_amplitude,
drift_frequency)
eval_dataset = torch.utils.data.TensorDataset(X_eval, y_eval)
eval_data_loader = torch.utils.data.DataLoader(eval_dataset, batch_size=10, shuffle=False)
# Experiment Loop
results = {}
input_points = {}
for k in k_values:
accuracies = []
input_points[k] = []
for run in range(num_runs):
model = SimpleNN(num_hidden_layers, hidden_layer_size)
optimizer = optim.SGD(model.parameters(), lr=0.01)
train_losses = []
current_run_points =
[]
generated_points_count
= 0
train_points_with_drift
= [] # Store training
points and their drifts
criterion = nn.BCEWithLogitsLoss()
while generated_points_count < N_desired:
generated_points_count
+= 1
if generated_points_count % k == 0:
x = torch.randn(1, 3) # Reset x to a new
random value here
x.requires_grad_(True)
current_run_points.append((x.detach().numpy(), 'blue'))
else:
x = x.detach().clone().requires_grad_(True)
with torch.no_grad():
supervisor_output = supervisor_function(x, generated_points_count, drift_amplitude,
drift_frequency)
model_output
= model(x)
loss_gradient_ascent
= criterion(model_output, supervisor_output.float().unsqueeze(1))
loss_gradient_ascent.backward()
if x.grad is not None:
x = x
+ eta_x * x.grad.detach()
current_run_points.append((x.detach().numpy(), 'red'))
# Training Step
optimizer.zero_grad()
supervisor_output
= supervisor_function(x, generated_points_count, drift_amplitude, drift_frequency)
model_output
= model(x)
loss = criterion(model_output, supervisor_output.float().unsqueeze(1))
loss.backward()
optimizer.step()
train_losses.append(loss.item())
# Store training
point and drift:
train_points_with_drift.append((generated_points_count,
x.detach().clone()))
input_points[k].append(current_run_points)
accuracy = calculate_accuracy(model, eval_data_loader, train_points_with_drift,
drift_amplitude, drift_frequency)
accuracies.append(accuracy)
# print(f'Run: {run + 1}, k: {k}, Accuracy: {accuracy:.2f}%')
results[k] = np.mean(accuracies)
print(f'k: {k}, Average
Accuracy: {results[k]:.2f}%')
# END OF THE CODE
=========================================
Summary of
experiments:
# Hyperparameters
eta_x = 0.01
k_values = list(range(1, 11, 1))
num_runs = 5
N_desired = X
drift_amplitude = Y
drift_frequency = Z
num_hidden_layers = 2
hidden_layer_size = 20
5000
N_desired = 5000
drift_amplitude = 0.0
drift_frequency = 0.0
k:
1, Average Accuracy: 91.78%
k: 2, Average Accuracy: 90.98%
k: 3, Average Accuracy: 89.44%
k: 4, Average Accuracy: 88.26%
k: 5, Average
Accuracy: 91.35%
k: 6, Average Accuracy: 89.93%
k: 7, Average Accuracy: 89.98%
k: 8, Average Accuracy: 91.76%
k: 9, Average Accuracy: 89.69%
k: 10, Average Accuracy:
90.89%
batch_size = 25
k:
1, Average Accuracy: 92.27%
k: 2, Average Accuracy: 89.19%
k: 3, Average Accuracy: 90.21%
k: 4, Average Accuracy: 90.27%
k: 5, Average Accuracy: 88.21%
k: 6, Average Accuracy: 91.41%
k: 7, Average Accuracy: 90.96%
k: 8, Average Accuracy: 90.26%
k: 9, Average Accuracy: 90.89%
k: 10, Average Accuracy: 91.61%
N_desired = 5000
drift_amplitude = 0.5
drift_frequency = 0.0001
k: 1, Average Accuracy: 90.70%
k: 2, Average Accuracy: 89.64%
k: 3, Average Accuracy: 84.40%
k: 4, Average Accuracy: 79.96%
k: 5,
Average Accuracy: 92.66%
k: 6, Average Accuracy: 90.92%
k: 7, Average Accuracy: 88.96%
k: 8, Average Accuracy: 87.98%
k: 9, Average Accuracy: 90.18%
k: 10, Average Accuracy: 86.32%
N_desired = 5000
drift_amplitude = 0.5
drift_frequency = 0.001
k: 1, Average Accuracy: 71.60%
k: 2, Average Accuracy: 78.40%
k:
3, Average Accuracy: 83.78%
k: 4, Average Accuracy: 77.70%
k: 5, Average Accuracy: 83.02%
k: 6, Average Accuracy: 69.96%
k: 7, Average Accuracy: 77.54%
k: 8, Average Accuracy: 81.10%
k: 9, Average Accuracy: 77.58%
k: 10, Average Accuracy:
70.90%
N_desired = 5000
drift_amplitude = 0.5
drift_frequency = 0.01
k: 1, Average Accuracy: 71.62%
k: 2, Average Accuracy: 69.42%
k: 3, Average Accuracy: 70.02%
k: 4, Average Accuracy: 74.20%
k:
5, Average Accuracy: 77.60%
k: 6, Average Accuracy: 69.92%
k: 7, Average Accuracy: 74.24%
k: 8, Average Accuracy: 73.58%
k: 9, Average Accuracy: 74.22%
k: 10, Average Accuracy:
73.50%
batch_size = 25
k: 1, Average Accuracy: 75.95%
k: 2, Average Accuracy: 74.65%
k: 3, Average Accuracy: 73.52%
k: 4, Average Accuracy: 71.07%
k: 5, Average Accuracy: 74.58%
k: 6, Average Accuracy: 72.07%
k: 7, Average Accuracy: 76.78%
k: 8, Average Accuracy: 72.48%
k: 9, Average Accuracy: 72.46%
k:
10, Average Accuracy: 77.01%
N_desired = 5000
drift_amplitude = 0.5
drift_frequency = 0.1
k: 1, Average Accuracy: 69.04%
k: 2, Average Accuracy: 71.82%
k:
3, Average Accuracy: 75.24%
k: 4, Average Accuracy: 67.14%
k: 5, Average Accuracy: 69.56%
k: 6, Average Accuracy: 72.38%
k: 7, Average Accuracy: 68.14%
k: 8, Average Accuracy: 71.52%
k: 9, Average Accuracy: 69.30%
k: 10, Average Accuracy:
67.66%
N_desired = 5000
drift_amplitude = 1.0
drift_frequency = 0.0001
k: 1, Average Accuracy: 68.28%
k: 2, Average Accuracy: 66.90%
k:
3, Average Accuracy: 75.18%
k: 4, Average Accuracy: 73.78%
k: 5, Average Accuracy: 69.60%
k: 6, Average Accuracy: 68.94%
k: 7, Average Accuracy: 64.86%
k: 8, Average Accuracy: 72.88%
k: 9, Average Accuracy: 68.10%
k: 10, Average Accuracy:
67.92%
N_desired = 5000
drift_amplitude = 1.0
drift_frequency = 0.001
k:
1, Average Accuracy: 62.24%
k: 2, Average Accuracy: 61.90%
k: 3, Average Accuracy: 61.66%
k: 4, Average Accuracy: 60.24%
k: 5, Average Accuracy: 59.32%
k: 6, Average Accuracy: 60.54%
k: 7, Average Accuracy: 60.50%
k: 8, Average Accuracy: 61.72%
k: 9, Average Accuracy: 62.00%
k: 10, Average Accuracy:
60.72%
N_desired = 5000
drift_amplitude = 1.0
drift_frequency = 0.01
k: 1, Average Accuracy: 55.02%
k: 2, Average Accuracy: 53.34%
k: 3, Average Accuracy: 56.14%
k: 4, Average Accuracy: 55.20%
k: 5, Average Accuracy: 57.98%
k:
6, Average Accuracy: 58.12%
k: 7, Average Accuracy: 54.22%
k: 8, Average Accuracy: 55.70%
k: 9, Average Accuracy: 55.72%
k: 10, Average Accuracy:
55.22%
N_desired = 5000
drift_amplitude = 1.0
drift_frequency = 0.1
k: 1, Average Accuracy: 54.58%
k: 2, Average Accuracy: 54.78%
k: 3, Average Accuracy: 55.92%
k: 4, Average Accuracy: 54.00%
k: 5, Average Accuracy: 55.56%
k: 6, Average Accuracy: 53.44%
k: 7, Average Accuracy: 54.78%
k: 8, Average Accuracy: 54.72%
k: 9, Average Accuracy: 56.14%
k:
10, Average Accuracy: 56.76%
N_desired = 5000
drift_amplitude = –0.5
drift_frequency = 0.0001
k: 1, Average Accuracy: 84.14%
k: 2, Average Accuracy: 74.42%
k: 3, Average Accuracy: 80.30%
k: 4, Average Accuracy: 74.92%
k: 5, Average Accuracy: 83.30%
k: 6,
Average Accuracy: 84.86%
k: 7, Average Accuracy: 82.10%
k: 8, Average Accuracy: 82.46%
k: 9, Average Accuracy: 73.68%
k: 10, Average Accuracy: 82.58%
N_desired = 5000
drift_amplitude = –0.5
drift_frequency = 0.001
k: 1, Average Accuracy: 79.32%
k: 2, Average Accuracy: 84.80%
k:
3, Average Accuracy: 86.26%
k: 4, Average Accuracy: 80.86%
k: 5, Average Accuracy: 81.48%
k: 6, Average Accuracy: 81.62%
k: 7, Average Accuracy: 78.96%
k: 8, Average Accuracy: 81.12%
k: 9, Average Accuracy: 81.30%
k: 10, Average Accuracy:
81.22%
N_desired = 5000
drift_amplitude = –0.5
drift_frequency = 0.01
k: 1, Average Accuracy: 69.84%
k: 2, Average Accuracy: 70.68%
k: 3, Average Accuracy: 65.06%
k: 4, Average Accuracy: 68.18%
k: 5, Average Accuracy: 69.50%
k: 6, Average Accuracy: 74.22%
k: 7, Average Accuracy: 73.72%
k: 8, Average Accuracy: 69.80%
k: 9, Average Accuracy: 66.84%
k:
10, Average Accuracy: 74.38%
N_desired = 5000
drift_amplitude = –0.5
drift_frequency = 0.1
k: 1, Average Accuracy: 74.50%
k: 2, Average Accuracy: 66.48%
k: 3, Average Accuracy: 70.50%
k: 4, Average Accuracy: 68.18%
k: 5, Average Accuracy: 69.26%
k: 6, Average Accuracy: 69.34%
k: 7, Average Accuracy: 71.50%
k:
8, Average Accuracy: 76.48%
k: 9, Average Accuracy: 69.28%
k: 10, Average Accuracy:
61.38%
20000
N_desired = 20000
drift_amplitude = 0.0
drift_frequency = 0.0
k:
1, Average Accuracy: 97.98%
k: 2, Average Accuracy: 97.94%
k: 3, Average Accuracy: 97.59%
k: 4, Average Accuracy: 97.47%
k: 5, Average Accuracy: 97.32%
k: 6, Average Accuracy: 97.18%
k: 7, Average Accuracy: 97.10%
k: 8, Average Accuracy: 96.90%
k: 9, Average Accuracy: 97.10%
k: 10, Average Accuracy:
96.35%
N_desired = 20000
drift_amplitude = 1.0
drift_frequency = 0.0001
k:
1, Average Accuracy: 62.72%
k: 2, Average Accuracy: 61.82%
k: 3, Average Accuracy: 62.50%
k: 4, Average Accuracy: 62.46%
k: 5, Average Accuracy: 62.22%
k: 6, Average Accuracy: 61.94%
k: 7, Average Accuracy: 61.86%
k: 8, Average Accuracy: 62.12%
k: 9, Average Accuracy: 62.46%
k: 10, Average Accuracy:
61.80%
N_desired = 20000
drift_amplitude = 1.0
drift_frequency = 0.001
k: 1, Average Accuracy: 64.36%
k: 2, Average Accuracy: 63.60%
k: 3, Average Accuracy: 64.22%
k: 4, Average Accuracy: 65.46%
k: 5, Average Accuracy: 64.92%
k: 6, Average Accuracy: 65.80%
k:
7, Average Accuracy: 65.82%
k: 8, Average Accuracy: 64.60%
k: 9, Average Accuracy: 65.66%
k: 10, Average Accuracy:
63.56%
N_desired = 20000
drift_amplitude = 1.0
drift_frequency = 0.01
k: 1, Average Accuracy: 62.10%
k: 2, Average Accuracy: 62.96%
k: 3, Average Accuracy: 63.02%
k: 4, Average Accuracy: 63.04%
k: 5, Average Accuracy: 61.86%
k: 6, Average Accuracy: 63.28%
k: 7, Average Accuracy: 62.58%
k: 8, Average Accuracy: 63.92%
k: 9, Average Accuracy: 64.14%
k:
10, Average Accuracy: 64.24%
N_desired = 20000
drift_amplitude = 1.0
drift_frequency = 0.1
k: 1, Average Accuracy: 62.04%
k: 2, Average Accuracy: 62.30%
k:
3, Average Accuracy: 62.98%
k: 4, Average Accuracy: 62.28%
k: 5, Average Accuracy: 62.24%
k: 6, Average Accuracy: 61.24%
k: 7, Average Accuracy: 61.46%
k: 8, Average Accuracy: 61.20%
k: 9, Average Accuracy: 62.78%
k: 10, Average Accuracy:
62.00%
N_desired = 20000
drift_amplitude = 0.5
drift_frequency = 0.0001
k: 1, Average Accuracy: 83.74%
k: 2, Average Accuracy: 82.66%
k: 3, Average Accuracy: 83.10%
k: 4, Average Accuracy: 82.78%
k: 5, Average Accuracy: 81.82%
k: 6, Average Accuracy: 83.42%
k: 7, Average Accuracy: 84.06%
k: 8, Average Accuracy: 79.56%
k:
9, Average Accuracy: 85.20%
k: 10, Average Accuracy:
80.44%
N_desired = 20000
drift_amplitude = 0.5
drift_frequency = 0.001
k: 1, Average Accuracy: 84.20%
k: 2, Average Accuracy: 84.56%
k: 3, Average Accuracy: 82.08%
k: 4, Average Accuracy: 84.10%
k: 5, Average Accuracy: 82.82%
k: 6, Average Accuracy: 83.70%
k: 7, Average Accuracy: 85.58%
k: 8, Average Accuracy: 83.08%
k: 9, Average Accuracy: 83.64%
k:
10, Average Accuracy: 86.20%
N_desired = 20000
drift_amplitude = 0.5
drift_frequency = 0.01
k: 1, Average Accuracy: 86.92%
k: 2, Average Accuracy: 86.26%
k: 3, Average Accuracy: 88.10%
k: 4, Average Accuracy: 88.88%
k: 5, Average Accuracy: 87.96%
k: 6, Average Accuracy: 90.06%
k: 7, Average Accuracy: 89.60%
k:
8, Average Accuracy: 90.66%
k: 9, Average Accuracy: 88.80%
k: 10, Average Accuracy:
88.88%
N_desired = 20000
drift_amplitude = 0.5
drift_frequency = 0.1
k: 1, Average Accuracy: 89.52%
k: 2, Average Accuracy: 89.32%
k: 3, Average Accuracy: 89.64%
k: 4, Average Accuracy: 88.56%
k: 5, Average Accuracy: 90.24%
k: 6, Average Accuracy: 90.12%
k: 7, Average Accuracy: 88.06%
k:
8, Average Accuracy: 90.34%
k: 9, Average Accuracy: 89.38%
k: 10, Average Accuracy:
89.60%
N_desired = 20000
drift_amplitude = –0.5
drift_frequency = 0.0001
k: 1, Average Accuracy: 79.68%
k: 2, Average Accuracy: 80.88%
k: 3, Average Accuracy: 81.06%
k: 4, Average Accuracy: 81.36%
k: 5, Average Accuracy: 80.52%
k:
6, Average Accuracy: 81.62%
k: 7, Average Accuracy: 79.70%
k: 8, Average Accuracy: 80.98%
k: 9, Average Accuracy: 78.20%
k: 10, Average Accuracy:
79.60%
N_desired = 20000
drift_amplitude = –0.5
drift_frequency = 0.001
k: 1, Average Accuracy: 77.86%
k: 2, Average Accuracy: 79.34%
k: 3, Average Accuracy: 78.36%
k: 4, Average Accuracy: 80.38%
k: 5, Average Accuracy: 80.38%
k: 6, Average Accuracy: 79.84%
k: 7, Average Accuracy: 78.04%
k: 8, Average Accuracy: 78.28%
k:
9, Average Accuracy: 80.60%
k: 10, Average Accuracy:
80.58%
N_desired = 20000
drift_amplitude = –0.5
drift_frequency = 0.01
k: 1, Average Accuracy: 90.48%
k: 2, Average Accuracy: 89.48%
k: 3, Average Accuracy: 89.14%
k: 4, Average Accuracy: 89.80%
k:
5, Average Accuracy: 91.16%
k: 6, Average Accuracy: 88.14%
k: 7, Average Accuracy: 89.40%
k: 8, Average Accuracy: 89.06%
k: 9, Average Accuracy: 88.36%
k: 10, Average Accuracy:
88.92%
N_desired = 20000
drift_amplitude = –0.5
drift_frequency = 0.01
k: 1, Average Accuracy: 89.08%
k: 2, Average Accuracy: 88.86%
k: 3, Average Accuracy: 88.00%
k: 4, Average Accuracy: 88.40%
k: 5, Average Accuracy: 88.34%
k: 6, Average Accuracy: 88.06%
k:
7, Average Accuracy: 90.14%
k: 8, Average Accuracy: 88.16%
k: 9, Average Accuracy: 89.66%
k: 10, Average Accuracy:
88.86%
500
N_desired = 500
drift_amplitude = 0.0
drift_frequency = 0.0
k: 1, Average Accuracy: 55.06%
k: 2, Average Accuracy: 51.89%
k: 3, Average Accuracy: 53.63%
k: 4, Average Accuracy: 53.74%
k: 5, Average Accuracy: 55.75%
k: 6, Average Accuracy: 53.31%
k: 7, Average Accuracy: 51.52%
k: 8, Average Accuracy: 48.12%
k:
9, Average Accuracy: 58.98%
k: 10, Average Accuracy:
48.94%
N_desired = 500
drift_amplitude = 1.0
drift_frequency = 0.0001
k: 1, Average Accuracy: 60.46%
k: 2, Average Accuracy: 61.97%
k: 3, Average Accuracy: 61.69%
k: 4, Average Accuracy: 60.49%
k: 5, Average Accuracy: 61.09%
k: 6, Average Accuracy: 61.37%
k: 7, Average Accuracy: 61.24%
k:
8, Average Accuracy: 62.08%
k: 9, Average Accuracy: 60.00%
k: 10, Average Accuracy:
58.47%
N_desired = 500
drift_amplitude = 1.0
drift_frequency = 0.001
k:
1, Average Accuracy: 64.00%
k: 2, Average Accuracy: 63.97%
k: 3, Average Accuracy: 63.98%
k: 4, Average Accuracy: 63.97%
k: 5, Average Accuracy: 63.86%
k: 6, Average Accuracy: 63.98%
k: 7, Average Accuracy: 63.90%
k: 8, Average Accuracy: 63.98%
k: 9, Average Accuracy: 63.90%
k: 10, Average Accuracy:
63.82%
N_desired = 500
drift_amplitude = 1.0
drift_frequency = 0.01
k: 1, Average Accuracy: 53.31%
k: 2, Average Accuracy: 53.29%
k: 3,
Average Accuracy: 53.34%
k: 4, Average Accuracy: 53.29%
k: 5, Average Accuracy: 53.33%
k: 7, Average Accuracy: 53.32%
k: 8, Average Accuracy: 53.29%
k: 9, Average Accuracy: 53.31%
k: 10, Average Accuracy: 53.34%
N_desired = 500
drift_amplitude = 1.0
drift_frequency = 0.1
k: 1, Average Accuracy: 50.60%
k: 2, Average Accuracy: 50.58%
k: 3, Average Accuracy: 50.56%
k:
4, Average Accuracy: 50.68%
k: 5, Average Accuracy: 50.58%
k: 6, Average Accuracy: 50.62%
k: 7, Average Accuracy: 50.53%
k: 8, Average Accuracy: 50.60%
k: 9, Average Accuracy: 50.51%
k: 10, Average Accuracy:
50.55%
N_desired = 500
drift_amplitude = 0.5
drift_frequency = 0.0001
k: 1, Average Accuracy: 57.68%
k: 2, Average Accuracy: 61.47%
k: 3, Average Accuracy: 61.00%
k: 4, Average Accuracy: 57.40%
k: 5, Average Accuracy: 57.04%
k:
6, Average Accuracy: 62.91%
k: 7, Average Accuracy: 61.16%
k: 8, Average Accuracy: 58.18%
k: 9, Average Accuracy: 56.64%
k: 10, Average Accuracy:
58.01%
N_desired = 500
drift_amplitude = 0.5
drift_frequency = 0.001
k: 1, Average Accuracy: 63.10%
k: 2, Average Accuracy: 63.00%
k: 3, Average Accuracy: 63.07%
k: 4, Average Accuracy: 63.05%
k:
5, Average Accuracy: 63.27%
k: 6, Average Accuracy: 63.22%
k: 7, Average Accuracy: 63.15%
k: 8, Average Accuracy: 63.01%
k: 9, Average Accuracy: 63.04%
k: 10, Average Accuracy:
63.08%
N_desired = 500
drift_amplitude = 0.5
drift_frequency = 0.01
k: 1, Average Accuracy: 53.06%
k: 2, Average Accuracy: 52.94%
k: 3, Average Accuracy: 52.96%
k: 4, Average Accuracy: 53.01%
k: 5, Average Accuracy: 53.04%
k:
6, Average Accuracy: 53.16%
k: 7, Average Accuracy: 53.06%
k: 8, Average Accuracy: 52.96%
k: 9, Average Accuracy: 52.81%
k: 10, Average Accuracy:
52.98%
N_desired = 500
drift_amplitude = 0.5
drift_frequency = 0.1
k: 1, Average Accuracy: 46.38%
k: 2, Average Accuracy: 46.35%
k: 3, Average Accuracy: 46.32%
k:
4, Average Accuracy: 46.47%
k: 5, Average Accuracy: 46.32%
k: 6, Average Accuracy: 46.34%
k: 7, Average Accuracy: 46.37%
k: 8, Average Accuracy: 46.40%
k: 9, Average Accuracy: 46.46%
k: 10, Average Accuracy:
46.28%
N_desired = 500
drift_amplitude = –0.5
drift_frequency = 0.0001
k: 1, Average Accuracy: 50.10%
k: 2, Average Accuracy: 43.84%
k: 3, Average Accuracy: 46.95%
k: 4, Average Accuracy: 46.24%
k: 5, Average Accuracy: 47.72%
k: 6, Average Accuracy: 48.60%
k: 7, Average Accuracy: 49.25%
k: 8, Average Accuracy: 43.30%
k: 9, Average Accuracy: 45.81%
k:
10, Average Accuracy: 50.24%
N_desired = 500
drift_amplitude = –0.5
drift_frequency = 0.001
k: 1, Average Accuracy: 35.84%
k: 2, Average Accuracy: 35.94%
k: 3, Average Accuracy: 35.81%
k: 4, Average Accuracy: 35.86%
k: 5, Average Accuracy: 35.82%
k:
6, Average Accuracy: 36.22%
k: 7, Average Accuracy: 35.89%
k: 8, Average Accuracy: 36.09%
k: 9, Average Accuracy: 35.76%
k: 10, Average Accuracy:
35.74%
N_desired = 500
drift_amplitude = –0.5
drift_frequency = 0.01
k: 1, Average Accuracy: 48.58%
k:
2, Average Accuracy: 48.67%
k: 3, Average Accuracy: 48.50%
k: 4, Average Accuracy: 48.63%
k: 5, Average Accuracy: 48.44%
k: 6, Average Accuracy: 48.61%
k: 7, Average Accuracy: 48.50%
k: 8, Average Accuracy: 48.51%
k: 9, Average Accuracy: 48.58%
k: 10, Average Accuracy:
48.30%
N_desired = 500
drift_amplitude = –0.5
drift_frequency = 0.1
k: 1, Average Accuracy: 48.20%
k: 2, Average Accuracy: 48.23%
k: 3, Average Accuracy: 48.34%
k: 4, Average Accuracy: 48.18%
k: 5, Average Accuracy: 48.20%
k:
6, Average Accuracy: 48.35%
k: 7, Average Accuracy: 48.21%
k: 8, Average Accuracy: 48.15%
k: 9, Average Accuracy: 48.16%
k: 10, Average Accuracy:
48.10%
Part III
(Cross-validation of cloning techniques: based on
hybrid exploitation-exploration sampling vs. based on random sampling)
=========================================
# BEGINNING OF THE CODE
import torch
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
# Define the Supervisor function
def supervisor(x):
if x.dim() == 1:
x = x.unsqueeze(0)
return torch.sin(torch.log(x[:, 0]**2 + x[:, 1]**2 + x[:, 2]**2 + 1))
# Define an enhanced neural network model
class EnhancedNN(nn.Module):
def __init__(self):
super(EnhancedNN, self).__init__()
self.fc1 = nn.Linear(3, 50)
self.fc2 = nn.Linear(50, 100)
self.fc3 = nn.Linear(100, 50)
self.fc4 = nn.Linear(50, 1)
self.tanh = nn.Tanh()
def forward(self, x):
x = self.tanh(self.fc1(x))
x = self.tanh(self.fc2(x))
x = self.tanh(self.fc3(x))
x = self.fc4(x)
return x
# Function to generate random input samples within [-1, 1] interval
def generate_random_input(batch_size=1):
return 2 * torch.rand(batch_size, 3) - 1
# Function to generate and collect data for adversarial learning
def generate_and_collect_data(model, optimizer, initial_input, random_steps, num_steps=400, learning_rate_inputs=0.05):
data_inputs = []
data_outputs = []
all_inputs = []
x_t = initial_input.clone().detach().requires_grad_(True)
for t in range(num_steps):
output = model(x_t)
loss = torch.abs(output
- supervisor(x_t)).sum()
optimizer.zero_grad()
loss.backward()
with torch.no_grad():
x_t += learning_rate_inputs * x_t.grad
if torch.any(x_t.abs() > 1):
x_t
= generate_random_input(1)
x_t = torch.clamp(x_t,
-1, 1)
data_inputs.append(x_t.detach().numpy().squeeze())
all_inputs.append(x_t.detach().numpy().squeeze())
data_outputs.append(supervisor(x_t).detach().numpy().squeeze())
if (t + 1) % random_steps == 0:
x_t = generate_random_input(1)
x_t = x_t.clone().detach().requires_grad_(True)
return np.array(data_inputs), np.array(data_outputs), np.array(all_inputs)
# Function to train a neural network
def train_nn(model, optimizer, data_inputs, data_outputs, num_epochs=100):
losses = []
loss_fn = nn.MSELoss()
inputs = torch.tensor(data_inputs, dtype=torch.float32)
targets = torch.tensor(data_outputs, dtype=torch.float32).unsqueeze(1)
for epoch in range(num_epochs):
optimizer.zero_grad()
outputs = model(inputs)
loss = loss_fn(outputs, targets)
loss.backward()
optimizer.step()
losses.append(loss.item())
return losses
# Function to plot with a fitted curve for NN1 and a horizontal line for
NN2 in the first plot
def plot_with_fit(x, y1, y2, degree, xlabel, ylabel, title, avg_nn2=False):
# Fit curve for NN1
poly = PolynomialFeatures(degree=degree)
x_poly = poly.fit_transform(x[:, np.newaxis])
model1 = LinearRegression().fit(x_poly, y1)
x_fit = np.linspace(x.min(), x.max(), 100)
x_fit_poly = poly.transform(x_fit[:, np.newaxis])
plt.plot(x_fit, model1.predict(x_fit_poly),
'b--', label='NN1 Fit')
if avg_nn2:
# Horizontal line for NN2 average loss
nn2_avg_loss = np.mean(y2)
plt.axhline(y=nn2_avg_loss,
color='g', linestyle='--', label='NN2 Avg Loss')
else:
# Fit curve for NN2
model2 = LinearRegression().fit(x_poly, y2)
plt.plot(x_fit, model2.predict(x_fit_poly),
'g--', label='NN2 Fit')
plt.xlabel(xlabel)
plt.ylabel(ylabel)
plt.title(title)
plt.legend()
# Main function to execute the experiment
def main():
nn1_final_losses = []
nn2_final_losses = []
nn1_test_losses = []
nn2_test_losses = []
nn1_on_nn2_losses = []
nn2_on_nn1_losses = []
random_steps_values = []
# Define the range of random_steps
to iterate over
MIN_RANDOM_STEPS = 1
MAX_RANDOM_STEPS = 400
NUM_PLOTS = 10
# Calculate the step increment to have NUM_PLOTS
values
step_increment = (MAX_RANDOM_STEPS
- MIN_RANDOM_STEPS) // (NUM_PLOTS - 1)
selected_random_steps =
[MIN_RANDOM_STEPS + i * step_increment
for i in range(NUM_PLOTS)]
fig, axes = plt.subplots(NUM_PLOTS,
3, figsize=(18, 6 * NUM_PLOTS))
for idx, random_steps
in enumerate(selected_random_steps):
nn1 = EnhancedNN()
nn2 = EnhancedNN()
optimizer_nn1 = optim.Adam(nn1.parameters(), lr=0.01)
optimizer_nn2 = optim.Adam(nn2.parameters(), lr=0.01)
initial_input = torch.tensor([[1.0, 1.0, 1.0]], dtype=torch.float32)
adv_data_inputs, adv_data_outputs, all_adv_inputs
= generate_and_collect_data(nn1, optimizer_nn1, initial_input,
random_steps, num_steps=400, learning_rate_inputs=0.05)
rand_data_inputs = np.array([generate_random_input().numpy().squeeze() for _ in range(400)])
rand_data_outputs = np.array([supervisor(torch.tensor(x).unsqueeze(0)).detach().numpy().squeeze()
for x in rand_data_inputs])
losses_nn1 = train_nn(nn1, optimizer_nn1, adv_data_inputs, adv_data_outputs,
num_epochs=100)
losses_nn2 = train_nn(nn2, optimizer_nn2, rand_data_inputs, rand_data_outputs,
num_epochs=100)
nn1_eval_on_nn2 = nn.MSELoss()(nn1(torch.tensor(rand_data_inputs, dtype=torch.float32)),
torch.tensor(rand_data_outputs,
dtype=torch.float32).unsqueeze(1)).item()
nn2_eval_on_nn1 = nn.MSELoss()(nn2(torch.tensor(adv_data_inputs, dtype=torch.float32)),
torch.tensor(adv_data_outputs,
dtype=torch.float32).unsqueeze(1)).item()
test_data_inputs = np.array([generate_random_input().numpy().squeeze() for _ in range(400)])
test_data_outputs = np.array([supervisor(torch.tensor(x).unsqueeze(0)).detach().numpy().squeeze()
for x in test_data_inputs])
test_inputs = torch.tensor(test_data_inputs,
dtype=torch.float32)
test_outputs = torch.tensor(test_data_outputs,
dtype=torch.float32).unsqueeze(1)
with torch.no_grad():
nn1.eval()
nn2.eval()
test_loss_nn1 = nn.MSELoss()(nn1(test_inputs), test_outputs).item()
test_loss_nn2 = nn.MSELoss()(nn2(test_inputs), test_outputs).item()
nn1_final_losses.append(losses_nn1[-1])
nn2_final_losses.append(losses_nn2[-1])
nn1_test_losses.append(test_loss_nn1)
nn2_test_losses.append(test_loss_nn2)
nn1_on_nn2_losses.append(nn1_eval_on_nn2)
nn2_on_nn1_losses.append(nn2_eval_on_nn1)
random_steps_values.append(random_steps)
print(f"Random Steps: {random_steps}")
print(f"NN1 Final
Loss: {losses_nn1[-1]}")
print(f"NN2 Final
Loss: {losses_nn2[-1]}")
print(f"NN1 Test
Loss: {test_loss_nn1}")
print(f"NN2 Test
Loss: {test_loss_nn2}")
print(f"NN1
Evaluated on NN2 Training Data: {nn1_eval_on_nn2}")
print(f"NN2
Evaluated on NN1 Training Data: {nn2_eval_on_nn1}")
print()
# Plotting the inputs generated for NN1 and NN2
for each random_steps value
ax = axes[idx]
ax[0].scatter(all_adv_inputs[:, 0], adv_data_outputs, color='b', label='NN1 Inputs')
ax[0].scatter(rand_data_inputs[:, 0], rand_data_outputs, color='g', label='NN2 Inputs')
ax[0].set_xlabel('Input Variable 1')
ax[0].set_ylabel('Supervisor Output')
ax[0].set_title(f'Random Steps: {random_steps} - Input 1 vs Supervisor
Output')
ax[0].legend()
ax[1].scatter(all_adv_inputs[:, 1], adv_data_outputs, color='b', label='NN1 Inputs')
ax[1].scatter(rand_data_inputs[:, 1], rand_data_outputs, color='g', label='NN2 Inputs')
ax[1].set_xlabel('Input Variable 2')
ax[1].set_ylabel('Supervisor Output')
ax[1].set_title(f'Random Steps: {random_steps} - Input 2 vs Supervisor
Output')
ax[1].legend()
ax[2].scatter(all_adv_inputs[:, 2], adv_data_outputs, color='b', label='NN1 Inputs')
ax[2].scatter(rand_data_inputs[:, 2], rand_data_outputs, color='g', label='NN2 Inputs')
ax[2].set_xlabel('Input Variable 3')
ax[2].set_ylabel('Supervisor Output')
ax[2].set_title(f'Random Steps: {random_steps} - Input 3 vs Supervisor
Output')
ax[2].legend()
plt.tight_layout()
plt.show()
plt.figure(figsize=(18, 6))
# Plot NN1 performance metrics vs. random_steps
plt.subplot(1, 3, 1)
plot_with_fit(np.array(random_steps_values),
np.array(nn1_final_losses), np.array(nn2_final_losses),
2, 'Random Steps', 'Loss', 'Final Loss Comparison', avg_nn2=True)
plt.subplot(1, 3, 2)
plot_with_fit(np.array(random_steps_values),
np.array(nn1_test_losses), np.array(nn2_test_losses),
2, 'Random Steps', 'Loss', 'Test Loss Comparison')
plt.subplot(1, 3, 3)
plot_with_fit(np.array(random_steps_values),
np.array(nn1_on_nn2_losses), np.array(nn2_on_nn1_losses),
2, 'Random Steps', 'Loss', 'NN1 Evaluated on NN2 Training Data Comparison')
plt.tight_layout()
plt.show()
if __name__ == '__main__':
main()
# END OF THE CODE
=========================================
Samples of
experiments:
Random Steps: 1
NN1 Final Loss: 0.0015317702200263739
NN2 Final Loss: 0.002190273255109787
NN1 Test Loss: 0.0013659027172252536
NN2 Test Loss: 0.00160881073679775
NN1 Evaluated on NN2 Training Data:
0.00135025754570961
NN2 Evaluated on NN1 Training Data:
0.0015236580511555076
Random Steps: 45
NN1 Final Loss: 0.0007287207990884781
NN2 Final Loss: 0.001434766105376184
NN1 Test Loss: 0.0009654824971221387
NN2 Test Loss: 0.001738809747621417
NN1 Evaluated on NN2 Training Data:
0.0010279713897034526
NN2 Evaluated on NN1 Training Data:
0.002187825506553054
Random Steps: 89
NN1 Final Loss: 0.0006188334664329886
NN2 Final Loss: 0.008179793134331703
NN1 Test Loss: 0.0008723001810722053
NN2 Test Loss: 0.004259792622178793
NN1 Evaluated on NN2 Training Data:
0.0007401613984256983
NN2 Evaluated on NN1 Training Data:
0.004841355141252279
Random Steps: 133
NN1 Final Loss: 0.0005814940086565912
NN2 Final Loss: 0.0013621969847008586
NN1 Test Loss: 0.008560857735574245
NN2 Test Loss: 0.00253803888335824
NN1 Evaluated on NN2 Training Data: 0.00750516215339303
NN2 Evaluated on NN1 Training Data:
0.0037006991915404797
Random Steps: 177
NN1 Final Loss: 0.0006486102938652039
NN2 Final Loss: 0.0027915334794670343
NN1 Test Loss: 0.007573459297418594
NN2 Test Loss: 0.0016477829776704311
NN1 Evaluated on NN2 Training Data:
0.00797099806368351
NN2 Evaluated on NN1 Training Data:
0.0017539695836603642
Random Steps: 221
NN1 Final Loss: 0.00046385452151298523
NN2 Final Loss: 0.00017359764024149626
NN1 Test Loss: 0.006228944286704063
NN2 Test Loss: 0.00023868458811193705
NN1 Evaluated on NN2 Training Data:
0.007080787792801857
NN2 Evaluated on NN1 Training Data:
0.000366831460269168
Random Steps: 265
NN1 Final Loss: 0.004163611680269241
NN2 Final Loss: 0.0003624846285674721
NN1 Test Loss: 0.0130416015163064
NN2 Test Loss: 0.0002323080407222733
NN1 Evaluated on NN2 Training Data:
0.012894359417259693
NN2 Evaluated on NN1 Training Data:
0.00037340016569942236
Random Steps: 309
NN1 Final Loss: 0.0005194011027924716
NN2 Final Loss: 0.00161657202988863
NN1 Test Loss: 0.0012549218954518437
NN2 Test Loss: 0.0015449917409569025
NN1 Evaluated on NN2 Training Data:
0.0012794297654181719
NN2 Evaluated on NN1 Training Data:
0.0018619279144331813
Random Steps: 353
NN1 Final Loss: 0.0010127393761649728
NN2 Final Loss: 0.00019206157594453543
NN1 Test Loss: 0.003346421755850315
NN2 Test Loss: 0.00022059964248910546
NN1 Evaluated on NN2 Training Data:
0.0041451831348240376
NN2 Evaluated on NN1 Training Data:
0.0001841532503021881
Random Steps: 397
NN1 Final Loss: 0.002777648624032736
NN2 Final Loss: 8.350084681296721e-05
NN1 Test Loss: 0.0023985402658581734
NN2 Test Loss: 0.00012479138968046755
NN1 Evaluated on NN2 Training Data:
0.0021055673714727163
NN2 Evaluated
on NN1 Training Data: 9.541634790366516e-05
Option with other supervisor
function and hyperparameters:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim.lr_scheduler
import ReduceLROnPlateau
import matplotlib.pyplot as plt
import numpy
as np
from scipy.stats import ttest_rel
import seaborn as sns
import pandas as pd
from math import pi
# Define the complex Supervisor function
def supervisor(x):
if x.dim()
== 1:
x = x.unsqueeze(0)
term1 = torch.sin(5 * x[:, 0]) * torch.cos(3 * x[:, 1])
term2 = torch.sin(3 * x[:, 1]) * torch.cos(5 * x[:, 2])
term3 = torch.sin(4 * x[:, 2]) * torch.cos(4 * x[:, 0])
term4 = x[:, 0]**2 - x[:, 1]**2 + x[:, 2]**2
return term1 + term2 + term3 + term4
# Define an enhanced neural network model
class EnhancedNN(nn.Module):
def __init__(self, dropout_prob=0.2, weight_decay=1e-5):
super(EnhancedNN, self).__init__()
self.fc1 = nn.Linear(3, 50)
self.fc2 = nn.Linear(50, 100)
self.fc3 = nn.Linear(100, 50)
self.fc4 = nn.Linear(50, 1)
self.tanh = nn.Tanh()
self.dropout = nn.Dropout(dropout_prob)
self.weight_decay = weight_decay
def forward(self, x):
x = self.tanh(self.fc1(x))
x = self.dropout(x)
x = self.tanh(self.fc2(x))
x = self.dropout(x)
x = self.tanh(self.fc3(x))
x = self.dropout(x)
x = self.fc4(x)
return x
def l2_regularization_loss(self):
l2_reg = torch.tensor(0., device=self.fc1.weight.device)
for name, param in self.named_parameters():
if 'weight' in name:
l2_reg += torch.norm(param,
p=2)
return self.weight_decay * l2_reg.item()
# Return as Python float
# Function to generate random input samples within
[-1, 1] interval
def generate_random_input(batch_size=1):
return 2 * torch.rand(batch_size, 3) - 1
# Function to train a neural network
def train_nn(model, optimizer, data_inputs, data_outputs, num_epochs=20):
losses = []
loss_fn = nn.MSELoss()
inputs = torch.tensor(data_inputs, dtype=torch.float32)
targets = torch.tensor(data_outputs, dtype=torch.float32).unsqueeze(1)
for epoch in range(num_epochs):
optimizer.zero_grad()
outputs = model(inputs)
loss = loss_fn(outputs, targets)
loss += model.l2_regularization_loss()
# Add L2 regularization loss
loss.backward()
optimizer.step()
losses.append(loss.item())
return losses
# Function to plot results
def plot_results(x, y1, y2, xlabel, ylabel, title, avg_nn2=False):
# Plot NN1 results
plt.plot(x,
y1, 'b--', label='NN1 Results')
if avg_nn2:
# Horizontal line
for NN2 average loss
nn2_avg_loss = np.mean(y2)
plt.axhline(y=nn2_avg_loss, color='g', linestyle='--', label='NN2 Avg Loss')
else:
# Plot NN2 results
plt.plot(x, y2, 'g--', label='NN2 Results')
plt.xlabel(xlabel)
plt.ylabel(ylabel)
plt.title(title)
plt.legend()
# Main function to execute the experiment with
iterative retraining
def main():
num_steps = 200
num_epochs = 2
initial_learning_rate
= 0.01
learning_rate_inputs
= 0.5
dropout_prob =
0.2
weight_decay =
1e-5
nn1_final_losses = []
nn2_final_losses = []
nn1_test_losses = []
nn2_test_losses = []
nn1_on_nn2_losses = []
nn2_on_nn1_losses = []
random_steps_values
= []
# Define the range
of random_steps to iterate over
selected_random_steps
= list(range(1, 11)) # Using the first
10 values of random steps: 1, 2, 3, ..., 10
fig, axes = plt.subplots(len(selected_random_steps), 3, figsize=(18, 6 * len(selected_random_steps)))
for idx, random_steps in enumerate(selected_random_steps):
nn1 = EnhancedNN(dropout_prob=dropout_prob, weight_decay=weight_decay)
nn2 = EnhancedNN(dropout_prob=dropout_prob, weight_decay=weight_decay)
optimizer_nn1 = optim.Adam(nn1.parameters(), lr=initial_learning_rate)
optimizer_nn2 = optim.Adam(nn2.parameters(), lr=initial_learning_rate)
scheduler_nn1 = ReduceLROnPlateau(optimizer_nn1, mode='min', patience=5)
scheduler_nn2 = ReduceLROnPlateau(optimizer_nn2, mode='min', patience=5)
data_inputs_nn1 = []
data_outputs_nn1 = []
data_inputs_nn2 = []
data_outputs_nn2 = []
initial_input
= torch.tensor([[1.0, 1.0, 1.0]], dtype=torch.float32)
x_t
= initial_input.clone().detach().requires_grad_(True)
for t in range(num_steps):
# Adversarial sampling for NN1
if t % random_steps != 0:
output = nn1(x_t)
loss = torch.abs(output
- supervisor(x_t)).sum()
optimizer_nn1.zero_grad()
loss.backward()
with torch.no_grad():
x_t += learning_rate_inputs
* x_t.grad
if torch.any(x_t.abs() > 1):
x_t = generate_random_input(1)
x_t = torch.clamp(x_t, -1, 1)
else:
x_t = generate_random_input(1)
# Add the new sample to the dataset for NN1
data_inputs_nn1.append(x_t.detach().numpy().squeeze())
data_outputs_nn1.append(supervisor(x_t).detach().numpy().squeeze())
# Retrain NN1 with the updated dataset
train_nn(nn1,
optimizer_nn1, np.array(data_inputs_nn1), np.array(data_outputs_nn1), num_epochs=num_epochs)
scheduler_nn1.step(train_nn(nn1, optimizer_nn1, np.array(data_inputs_nn1),
np.array(data_outputs_nn1), num_epochs=num_epochs)[-1])
# Add a random sample to the dataset for NN2
random_input = generate_random_input(1)
data_inputs_nn2.append(random_input.numpy().squeeze())
data_outputs_nn2.append(supervisor(random_input).detach().numpy().squeeze())
# Retrain NN2 with the updated dataset
train_nn(nn2,
optimizer_nn2, np.array(data_inputs_nn2), np.array(data_outputs_nn2), num_epochs=num_epochs)
scheduler_nn2.step(train_nn(nn2, optimizer_nn2, np.array(data_inputs_nn2),
np.array(data_outputs_nn2), num_epochs=num_epochs)[-1])
x_t = x_t.clone().detach().requires_grad_(True)
# Convert lists to
numpy arrays
data_inputs_nn1_np = np.array(data_inputs_nn1)
data_outputs_nn1_np = np.array(data_outputs_nn1)
data_inputs_nn2_np = np.array(data_inputs_nn2)
data_outputs_nn2_np = np.array(data_outputs_nn2)
# Evaluate models
nn1_eval_on_nn2 = nn.MSELoss()(nn1(torch.tensor(data_inputs_nn2_np, dtype=torch.float32)),
torch.tensor(data_outputs_nn2_np,
dtype=torch.float32).unsqueeze(1)).item()
nn2_eval_on_nn1 = nn.MSELoss()(nn2(torch.tensor(data_inputs_nn1_np, dtype=torch.float32)),
torch.tensor(data_outputs_nn1_np,
dtype=torch.float32).unsqueeze(1)).item()
test_data_inputs
= np.array([generate_random_input().numpy().squeeze()
for _ in range(200)])
test_data_outputs
= np.array([supervisor(torch.tensor(x).unsqueeze(0)).detach().numpy().squeeze() for x in test_data_inputs])
test_inputs
= torch.tensor(test_data_inputs, dtype=torch.float32)
test_outputs
= torch.tensor(test_data_outputs, dtype=torch.float32).unsqueeze(1)
with torch.no_grad():
nn1.eval()
nn2.eval()
test_loss_nn1 = nn.MSELoss()(nn1(test_inputs), test_outputs).item()
test_loss_nn2 = nn.MSELoss()(nn2(test_inputs), test_outputs).item()
nn1_final_losses.append(train_nn(nn1,
optimizer_nn1, np.array(data_inputs_nn1), np.array(data_outputs_nn1), num_epochs=num_epochs)[-1])
nn2_final_losses.append(train_nn(nn2,
optimizer_nn2, np.array(data_inputs_nn2), np.array(data_outputs_nn2), num_epochs=num_epochs)[-1])
nn1_test_losses.append(test_loss_nn1)
nn2_test_losses.append(test_loss_nn2)
nn1_on_nn2_losses.append(nn1_eval_on_nn2)
nn2_on_nn1_losses.append(nn2_eval_on_nn1)
random_steps_values.append(random_steps)
print(f"Random Steps: {random_steps}")
print(f"NN1 Final
Loss: {nn1_final_losses[-1]}")
print(f"NN2 Final
Loss: {nn2_final_losses[-1]}")
print(f"NN1 Test
Loss: {test_loss_nn1}")
print(f"NN2 Test
Loss: {test_loss_nn2}")
print(f"NN1
Evaluated on NN2 Training Data: {nn1_eval_on_nn2}")
print(f"NN2
Evaluated on NN1 Training Data: {nn2_eval_on_nn1}")
print()
# Plotting the
inputs generated for NN1 and NN2 for each random_steps
value
ax = axes[idx]
ax[0].scatter(np.array(data_inputs_nn1)[:, 0], np.array(data_outputs_nn1),
color='b', label='NN1 Inputs')
ax[0].scatter(np.array(data_inputs_nn2)[:, 0], np.array(data_outputs_nn2),
color='g', label='NN2 Inputs')
ax[0].set_xlabel('Input Variable 1')
ax[0].set_ylabel('Supervisor Output')
ax[0].set_title(f'Random Steps: {random_steps} - Input 1 vs Supervisor Output')
# Add the actual
supervisor function as a red line
input_range
= np.linspace(-1, 1, 100)
supervisor_outputs
= [supervisor(torch.tensor([x,
0, 0], dtype=torch.float32)).item() for x in input_range]
ax[0].plot(input_range, supervisor_outputs, 'r-', linewidth=2, label='Supervisor Function')
ax[0].legend()
ax[1].scatter(np.array(data_inputs_nn1)[:, 1], np.array(data_outputs_nn1),
color='b', label='NN1 Inputs')
ax[1].scatter(np.array(data_inputs_nn2)[:, 1], np.array(data_outputs_nn2),
color='g', label='NN2 Inputs')
ax[1].set_xlabel('Input Variable 2')
ax[1].set_ylabel('Supervisor Output')
ax[1].set_title(f'Random Steps: {random_steps} - Input 2 vs Supervisor Output')
# Add the actual
supervisor function as a red line
supervisor_outputs
= [supervisor(torch.tensor([0, x, 0], dtype=torch.float32)).item() for x in input_range]
ax[1].plot(input_range, supervisor_outputs, 'r-', linewidth=2, label='Supervisor Function')
ax[1].legend()
ax[2].scatter(np.array(data_inputs_nn1)[:, 2], np.array(data_outputs_nn1),
color='b', label='NN1 Inputs')
ax[2].scatter(np.array(data_inputs_nn2)[:, 2], np.array(data_outputs_nn2),
color='g', label='NN2 Inputs')
ax[2].set_xlabel('Input Variable 3')
ax[2].set_ylabel('Supervisor Output')
ax[2].set_title(f'Random Steps: {random_steps} - Input 3 vs Supervisor Output')
# Add the actual
supervisor function as a red line
supervisor_outputs
= [supervisor(torch.tensor([0, 0, x], dtype=torch.float32)).item() for x in input_range]
ax[2].plot(input_range, supervisor_outputs, 'r-', linewidth=2, label='Supervisor Function')
ax[2].legend()
plt.tight_layout()
plt.show()
# Statistical
analysis: Paired t-test
t_test_results
= ttest_rel(nn1_final_losses, nn2_final_losses)
print(f"T-test results: Statistic={t_test_results.statistic}, p-value={t_test_results.pvalue}")
# Final plots comparing
performance for different random step values
plt.figure(figsize=(14, 8))
plt.subplot(2, 2, 1)
plot_with_spline(np.array(random_steps_values),
np.array(nn1_final_losses), np.array(nn2_final_losses),
xlabel='Random Steps', ylabel='Final Loss',
title='Final Training Loss vs Random
Steps')
plt.subplot(2, 2, 2)
plot_with_spline(np.array(random_steps_values),
np.array(nn1_test_losses), np.array(nn2_test_losses),
xlabel='Random Steps', ylabel='Test Loss',
title='Test Loss vs Random Steps', avg_nn2=True)
plt.subplot(2, 2, 3)
plot_with_spline(np.array(random_steps_values),
np.array(nn1_on_nn2_losses), np.array(nn2_on_nn1_losses),
xlabel='Random Steps', ylabel='Cross Evaluation Loss',
title='NN1 Evaluated on NN2 Training
Data vs Random Steps')
plt.subplot(2, 2, 4)
plot_with_spline(np.array(random_steps_values),
np.array(nn2_on_nn1_losses), np.array(nn1_on_nn2_losses),
xlabel='Random Steps', ylabel='Cross Evaluation Loss',
title='NN2 Evaluated on NN1 Training
Data vs Random Steps')
plt.tight_layout()
plt.show()
# Loss
distributions visualization using box plots and KDE plots
plt.figure(figsize=(14, 8))
plt.subplot(1, 2, 1)
sns.boxplot(data=[nn1_final_losses,
nn2_final_losses], palette="Set2")
plt.xticks([0, 1], ['NN1', 'NN2'])
plt.title('Box Plot of Final Losses')
plt.subplot(1, 2, 2)
sns.kdeplot(nn1_final_losses,
shade=True, color="b", label='NN1')
sns.kdeplot(nn2_final_losses,
shade=True, color="g", label='NN2')
plt.title('KDE Plot of Final Losses')
plt.legend()
plt.tight_layout()
plt.show()
# Comprehensive
radar chart for NN1 and NN2
categories = ['Final Loss', 'Test Loss', 'NN1 on NN2 Loss', 'NN2 on NN1
Loss']
nn1_stats = [np.mean(nn1_final_losses), np.mean(nn1_test_losses),
np.mean(nn1_on_nn2_losses), np.mean(nn2_on_nn1_losses)]
nn2_stats = [np.mean(nn2_final_losses), np.mean(nn2_test_losses),
np.mean(nn1_on_nn2_losses), np.mean(nn2_on_nn1_losses)]
df = pd.DataFrame({
'categories': categories,
'NN1': nn1_stats,
'NN2': nn2_stats
})
num_vars = len(categories)
angles = np.linspace(0, 2 * np.pi,
num_vars, endpoint=False).tolist()
angles += angles[:1]
fig, ax = plt.subplots(figsize=(8, 8), subplot_kw=dict(polar=True))
ax.set_theta_offset(pi / 2)
ax.set_theta_direction(-1)
plt.xticks(angles[:-1], categories)
ax.plot(angles,
nn1_stats + nn1_stats[:1], linewidth=1, linestyle='solid', label='NN1')
ax.fill(angles,
nn1_stats + nn1_stats[:1], alpha=0.25)
ax.plot(angles,
nn2_stats + nn2_stats[:1], linewidth=1, linestyle='solid', label='NN2')
ax.fill(angles,
nn2_stats + nn2_stats[:1], alpha=0.25)
plt.legend(loc='upper right', bbox_to_anchor=(0.1, 0.1))
plt.title('Performance Comparison of NN1 and NN2')
plt.show()
if __name__ == '__main__':
main()
Random Steps: 1
NN1 Final Loss: 1.3384140729904175
NN2 Final Loss: 1.107669472694397
NN1 Test Loss: 1.5605266094207764
NN2 Test Loss: 1.346555471420288
NN1 Evaluated on NN2 Training Data: 1.4067219495773315
NN2 Evaluated on NN1 Training Data: 0.9610530138015747
Random Steps: 2
NN1 Final Loss: 1.06387460231781
NN2 Final Loss: 1.4997347593307495
NN1 Test Loss: 0.8801223635673523
NN2 Test Loss: 1.5127872228622437
NN1 Evaluated on NN2 Training Data: 1.1085138320922852
NN2 Evaluated on NN1 Training Data: 1.6792957782745361
Random Steps: 3
NN1 Final Loss: 1.3806229829788208
NN2 Final Loss: 0.9313411116600037
NN1 Test Loss: 1.4325741529464722
NN2 Test Loss: 1.0685549974441528
NN1 Evaluated on NN2 Training Data: 1.1802006959915161
NN2 Evaluated on NN1 Training Data: 1.0751830339431763
Random Steps: 4
NN1 Final Loss: 1.3116343021392822
NN2 Final Loss: 1.2111485004425049
NN1 Test Loss: 1.441733956336975
NN2 Test Loss: 1.1786025762557983
NN1 Evaluated on NN2 Training Data: 1.4310415983200073
NN2 Evaluated on NN1 Training Data: 1.071882963180542
Random Steps: 5
NN1 Final Loss: 1.1880842447280884
NN2 Final Loss: 0.9008605480194092
NN1 Test Loss: 1.2661417722702026
NN2 Test Loss: 1.0851874351501465
NN1 Evaluated on NN2 Training Data: 1.0928021669387817
NN2 Evaluated on NN1 Training Data: 1.0507330894470215
Random Steps: 6
NN1 Final Loss: 1.1654314994812012
NN2 Final Loss: 1.0105348825454712
NN1 Test Loss: 1.180321455001831
NN2 Test Loss: 1.0429428815841675
NN1 Evaluated on NN2 Training Data: 1.3542147874832153
NN2 Evaluated on NN1 Training Data: 1.2956361770629883
Random Steps: 7
NN1 Final Loss: 0.971307635307312
NN2 Final Loss: 1.4175305366516113
NN1 Test Loss: 0.9469289183616638
NN2 Test Loss: 1.9934295415878296
NN1 Evaluated on NN2 Training Data: 0.9630601406097412
NN2 Evaluated on NN1 Training Data: 2.0200109481811523
Random Steps: 8
NN1 Final Loss: 1.1861439943313599
NN2 Final Loss: 1.1506813764572144
NN1 Test Loss: 1.061588168144226
NN2 Test Loss: 1.1636338233947754
NN1 Evaluated on NN2 Training Data: 1.0922435522079468
NN2 Evaluated on NN1 Training Data: 1.4341835975646973
Random Steps: 9
NN1 Final Loss: 1.2347348928451538
NN2 Final Loss: 1.0016813278198242
NN1 Test Loss: 1.4563069343566895
NN2 Test Loss: 1.0632199048995972
NN1 Evaluated on NN2 Training Data: 1.1518330574035645
NN2 Evaluated on NN1 Training Data: 1.1747355461120605
Random Steps: 10
NN1 Final Loss: 1.1127028465270996
NN2 Final Loss: 1.0497362613677979
NN1 Test Loss: 1.184678316116333
NN2 Test Loss: 1.1849241256713867
NN1 Evaluated on NN2 Training Data: 1.041130542755127
NN2 Evaluated on NN1 Training
Data: 1.195894718170166