FAERY

Introduction

An implementation of the FAERY [1] meta-learning algorithm for divergent search to the Metaworld benchmark [2]. Consists in a multi-objective optimization of a meta-population evaluated in a given number of evoluationary processes ran on parallelized environments.

Full project report available here: REPORT_ELRHARBI-FLEURY_YANNIS.pdf

For a meta-individual $p_j \text{ evaluated on } M \text{ environments in which it produced the }\zeta_{ij}$ lineage, it is given two scores :

with $\psi_i(s)$ a binary operator checking if $s$ solved the environment and $d_m(s, p_j)$ the expected number of mutations to reach $s$ from $p_j$ (approximated by its depth in the genealogy tree).

File structure

We strive to offer a modular implementation of the algorithm, hence each folder implements a package of classes :

• problem implements Metaworld agents as well as problem samplers
• novelty_search implements examples for the inner algorithm, here Novelty Search [3] and Quality Diversity [4].
• meta_learning implements a base class for meta-learning algorithms, in particular an implementation of FAERY
• gridworld toy environment, standalone implementation of FAERY, work in progress
• results contains output data, as well as python scripts that extract results and plot graphs

How to run the implementation

Python 3.6+, requirements

We included an example script able to launch FAERY with base parameters on different environments. It takes as argument the desired environment (assembly, basketball, buttonpress, hammer, and others if you create the corresponding base_{env}.params file) and outputs the results in the results folder.

To run a quick test and check that everything works, run the following command from the FAERY directory :

# To run a quick test:
./launchers/FAERY.sh test
# To run FAERY on Metaworld hammer:
./launchers/FAERY.sh hammer

During each meta-step, the solvers are saved as they are found in results/data. Then, the final evolution tables and meta-population's scores $f_0$ and $f_1$ are computed and saved for the given meta-step. Therefore, once the algorithm has ran, you should be able to extract and compile them as performance graphs or T-SNE plots by running the provided scripts in extraction scripts :

Set up the extraction

In order to plot the above graphs, one needs to set up the scripts with the provided .json files, the parameters of which are self explanatory. Please note that by setting some booleans, it is possible to enable the plotting of supplementary graphs, such as :

• a comparison graph for data obtained with different parameters

• an animation of the pareto front

• a TSNE plot for different perplexity parameters

• an animation of the solutions within the TSNE plot as they are found

References

[1] A. Salehi, A. Coninx and S. Doncieux, "Few-Shot Quality-Diversity Optimization," in IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4424-4431, April 2022, doi: 10.1109/LRA.2022.3148438.

[2] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning, Tianhe Yu and Deirdre Quillen and Zhanpeng He and Ryan Julian and Karol Hausman and Chelsea Finn and Sergey Levine, Conference on Robot Learning (CoRL) 2019

[3] Stephane Doncieux, Alban Laflaquière, Alexandre Coninx. Novelty search: a Theoretical Perspective. GECCO ’19: Genetic and Evolutionary Computation Conference, Jul 2019, Prague Czech Republic, France. pp.99-106, 10.1145/3321707.3321752. hal-02561846

[4] A. Cully and Y. Demiris, "Quality and Diversity Optimization: A Unifying Modular Framework," in IEEE Transactions on Evolutionary Computation, vol. 22, no. 2, pp. 245-259, April 2018, doi: 10.1109/TEVC.2017.2704781.