name: title class: title, middle ## GFlowNets ### Introduction and Applications to AI-Driven Scientific Discovery Alex Hernández-García (he/il/él) .turquoise[[MMICCs Webinar Series, SEA-CROGS](https://www.pnnl.gov/projects/sea-crogs) · March 28th 2023] .center[ <a href="https://mila.quebec/"><img src="../assets/images/slides/logos/mila-beige.png" alt="Mila" style="height: 4em"></a>      <a href="https://www.umontreal.ca/"><img src="../assets/images/slides/logos/udem-white.png" alt="UdeM" style="height: 4em"></a> ] .smaller[.footer[ Slides: [alexhernandezgarcia.github.io/slides/mmiccs-mar23](https://alexhernandezgarcia.github.io/slides/mmiccs-mar23) ]] --- ## Collaborators .left-column[ * Yoshua Bengio (Mila, UdeM) * Moksh Jain (Mila) * Emmanuel Bengio (Mila) * Kolya Malkin (Mila) * ... ] .right-column[ * Lena Simine (McGill) * Michael Kilgour (NYU) * Alexandra Volokhoval (Mila) * Nikita Saxena (Mila) * ... ] --- ## Outline * Part 1: Motivation: Why scientific discovery? * Challenges, limitations and opportunities for machine learning * Part 2: A gentle introduction to GFlowNets * Part 3: Applications to scientific discovery --- name: title class: title, middle ## Motivation: Why scientific discovery? ### Part 1 .center[] --- ## Why scientific discovery? .context[Climate change is a major challenge for humanity.] .left-column-66[.center[ <figure> <img src="../assets/images/slides/climatechange/anthropogenic_temperature_rise.png" alt="Historical global average temperature and the influence of modern humans" style="width: 90%"> <figcaption>.smaller[Modelled and observed global average temperatures in the last 2 millenia (source graphic: <a href="https://www.theguardian.com/science/2021/aug/09/humans-have-caused-unprecedented-and-irreversible-change-to-climate-scientists-warn">The Guardian</a>.)]</figcaption> </figure> ]] .right-column-33[ Consequences: * Melting glaciers and polar ice * Sea level rise * Heatwaves * Floods * Droughts * Wildfires * ... ] ??? * Flash floods kill **5,000** people per year. * Sea levels are expected to rise by **2 metres** by the end of the century * Rising sea levels could disrupt the lives of **1 billion people** by the end of 2050. * As much as **40% of the Amazon** forest is at risk of becoming a savanna. * In 2015, forest fires claimed roughly **980 000 $km^2$** of the world’s forest. * Forest fires emmitted **~1.8 Gt of CO2** in 2019. --- ## Why scientific discovery? .context[Climate change is a major challenge for humanity.] .center[ <figure> <img src="../assets/images/slides/climatechange/ipcc_scenarios.png" alt="IPCC 2022 - Scenarios" style="width: 60%"> <figcaption>Median global warming across modelled scenarios. Adapted from IPCC Sixth Assessment Report, 2022</figcaption> </figure> ] -- .conclusion["The evidence is clear: the time for action is now." .smaller[IPCC Sixth Assessment Report, 2022]] ??? * Category C1: scenarios that limit warming to 1.5°C in 2100 with a likelihood of greater than 50%, and reach or exceed warming of 1.5°C during the 21st century with a likelihood of 67% or less. * Category C2: same as C1 but exceed warming of 1.5°C during the 21st century with a likelihood of _greater_ than 67%. * Category C3: scenarios that limit peak warming to 2°C throughout the 21st century with a likelihood of greater than 67% * Category C8: scenarios that exceed warming of 4°C during the 21st century with a likelihood of 50% or greater. --- ## Why scientific discovery? .context["The time for action is now"] > "Limiting global warming will require major transitions in the energy sector. This will involve a substantial reduction in fossil fuel use, widespread electrification, .highlight1[improved energy efficiency, and use of alternative fuels (such as hydrogen)]." .cite[IPCC Sixth Assessment Report, 2022] > "Net-zero CO2 emissions from the industrial sector are challenging but possible. Reducing industry emissions will entail coordinated action throughout value chains to promote all mitigation options, including demand management, .highlight1[energy and materials efficiency, circular material flows], as well as abatement technologies and transformational changes in production processes." .cite[IPCC Sixth Assessment Report, 2022] -- <br> .conclusion[Mitigation of the climate crisis requires transformational changes in the energy and materials efficiency.] --- ## Why scientific discovery? ### The potential of better materials .context[The climate crisis demands more efficient materials.] * Improving material efficiency can reduce 0.93 ($\pm$ 0.23) GtCO₂-eq per year. * Fuel switching can reduce 2.1 ($\pm$ 0.52) GtCO₂-eq per year, only in the industry sector. * Carbon capture and storage can reduce 0.54 ($\pm$ 0.27) GtCO₂-eq per year in the energy sector. .right[.cite[IPCC Sixth Assessment Report (2022)]] .smaller[.footnote[† Global anthropogenic emissions in 2019 were estimated in 59 ($\pm$ 6.6) GtCO₂-eq. The budget from 2020 to limit warming to 1.5°C is estimated in 510 ($\pm$ 180) GtCO₂-eq.]] --- count: false ## Why scientific discovery? ### The potential of better materials .context[The climate crisis demands more efficient materials.] * Improving material efficiency can reduce 0.93 ($\pm$ 0.23) GtCO₂-eq per year. * Fuel switching can reduce 2.1 ($\pm$ 0.52) GtCO₂-eq per year, only in the industry sector. * Carbon capture and storage can reduce 0.54 ($\pm$ 0.27) GtCO₂-eq per year in the energy sector. .right[.cite[IPCC Sixth Assessment Report (2022)]] What are better, new materials needed for? * Electrocatalysts for fuel cells, hydrogen storage, industrial chemical reactions, carbon capture, etc. * Solid electrolytes for batteries. * Thin film materials for photovoltaics. * ... .smaller[.footnote[† Global anthropogenic emissions in 2019 were estimated in 59 ($\pm$ 6.6) GtCO₂-eq. The budget from 2020 to limit warming to 1.5°C is estimated in 510 ($\pm$ 180) GtCO₂-eq.]] --- ## Why scientific discovery? ### Scientific discoveries in history .context[Material discovery is a key ingredient for climate change mitigation.] -- Many notable scientific discoveries have occurred due to .highlight1[serendipity] or .highlight1[by accident]: -- * **Dynamite** (Alfred Nobel, 1867) * **X-rays** (Wilhelm C. Röntgen, 1895) * **Radioactivity** (Henri Becquerel and Marie Skłodowska–Curie, 1896) * **Penicillin** (Alexander Fleming, 1929) * **Cyanoacrylate (superglue)** (Harry Coover, 1942) * **Lysergic acid diethylamide (LSD)** (Albert Hofmann, 1943) -- <br> .conclusion[Clearly, we should not rely on serendipity to fight climate change.] ??? Joke experience with some of them, like penicillin and superglue. --- count: false ## Why scientific discovery? ### Scientific discoveries in history .context[Material discovery is a key ingredient for climate change mitigation.] .center[ <figure> <img src="../assets/images/slides/materials/paradigms_scientific_discovery_0.png" alt="Four paradigms of concrete science: empirical, theoretical, computational, and data-driven." style="width: 65%"> <figcaption>Four paradigms in scientific discovery. Source: <a href="https://www.nature.com/articles/s41524-022-00810-x">Li et al., 2022</a>. (<a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>)</figcaption> </figure> ] .references[ * Li et al. [Machine learning in concrete science: applications, challenges, and best practices](https://www.nature.com/articles/s41524-022-00810-x). Nature npj Computational Materials, 2022 ] ??? Example of concrete: most prevalent human-made material on Earth, and the most consumed commodity after water. The annual consumption of concrete in the world has reached 35 billion tons, which is twice as much as that of all other building materials combined. --- count: false ## Why scientific discovery? ### Scientific discoveries in history .context[Material discovery is a key ingredient for climate change mitigation.] .center[ <figure> <img src="../assets/images/slides/materials/paradigms_scientific_discovery_1.png" alt="Four paradigms of concrete science: empirical, theoretical, computational, and data-driven." style="width: 65%"> <figcaption>Four paradigms in scientific discovery. Source: <a href="https://www.nature.com/articles/s41524-022-00810-x">Li et al., 2022</a>. (<a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>)</figcaption> </figure> ] .references[ * Li et al. [Machine learning in concrete science: applications, challenges, and best practices](https://www.nature.com/articles/s41524-022-00810-x). Nature npj Computational Materials, 2022 ] ??? Concrete: The properties and performance of concrete can be tailored to meet design requirements by varying the type and quantity of the mixture constituents (e.g., cement, water, aggregate, and admixtures). Traditional approaches for designing concrete mixtures often rely on trial-and-error, iterative proportioning, processing, and characterization until the target properties are achieved. --- count: false ## Why scientific discovery? ### Scientific discoveries in history .context[Material discovery is a key ingredient for climate change mitigation.] .center[ <figure> <img src="../assets/images/slides/materials/paradigms_scientific_discovery_2.png" alt="Four paradigms of concrete science: empirical, theoretical, computational, and data-driven." style="width: 65%"> <figcaption>Four paradigms in scientific discovery. Source: <a href="https://www.nature.com/articles/s41524-022-00810-x">Li et al., 2022</a>. (<a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>)</figcaption> </figure> ] .references[ * Li et al. [Machine learning in concrete science: applications, challenges, and best practices](https://www.nature.com/articles/s41524-022-00810-x). Nature npj Computational Materials, 2022 ] ??? Concrete: it is possible to optimize the compressive strength of concrete mixtures by adjusting the water/cement ratio, total aggregate/cement ratio, and coarse aggregate/total aggregate ratio6. Yet the practical application of this iterative refinement approach is limited by the exponential increase in the number of specimens and experiments when complex concrete mixtures are studied and several compositional parameters are simultaneously considered as combinatorial variables. As a result, materials development in concrete science involves time-consuming validation/development cycles from laboratory trials to field applications. Efforts to accelerate knowledge acquisition and materials design in concrete science are thus of paramount importance. Beginning in the 1980s, the development of microstructural models of cement hydration has enabled a fundamental understanding of microstructure–property relationships in concrete7, which has marked the second paradigm. By applying basic laws of kinetics, thermodynamics, and mechanics, and providing analytical solutions to cement hydration. Successful demonstrations include the three-dimensional cement hydration and microstructure development model (CEMHYD3D)8,9; the hydration, morphology, and structural development model (HYMOSTRUC)10; the integrated particle kinetics model11; and the microstructural modeling platform (μic) --- count: false ## Why scientific discovery? ### Scientific discoveries in history .context[Material discovery is a key ingredient for climate change mitigation.] .center[ <figure> <img src="../assets/images/slides/materials/paradigms_scientific_discovery_3.png" alt="Four paradigms of concrete science: empirical, theoretical, computational, and data-driven." style="width: 65%"> <figcaption>Four paradigms in scientific discovery. Source: <a href="https://www.nature.com/articles/s41524-022-00810-x">Li et al., 2022</a>. (<a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>)</figcaption> </figure> ] .references[ * Li et al. [Machine learning in concrete science: applications, challenges, and best practices](https://www.nature.com/articles/s41524-022-00810-x). Nature npj Computational Materials, 2022 ] ??? Concrete: However, the complex nature of cement hydration makes it challenging to develop accurate and generalizable models, and these modeling approaches, to varying degrees, rely on thermochemical, physical, and structural data that must be obtained either from accurate experimental observations or from calculations at the atomistic and molecular scales. In this context, the use of density-functional theory (DFT) and classical molecular dynamics (MD) simulations has been explored in concrete science since the 2000s owing to the ever-growing computing power16. This has given rise to the third paradigm (computational science; Fig. 1), where the first-principle models have been integrated and employed to further describe cementitious materials properties and improve understanding of cement hydration. Related simulation efforts have focused primarily on cementitious phases such as the calcium silicate hydrate (C-S-H) gel, the essential reaction product of cement hydration. --- count: false ## Why scientific discovery? ### Scientific discoveries in history .context[Material discovery is a key ingredient for climate change mitigation.] .center[ <figure> <img src="../assets/images/slides/materials/paradigms_scientific_discovery_4.png" alt="Four paradigms of concrete science: empirical, theoretical, computational, and data-driven." style="width: 65%"> <figcaption>Four paradigms in scientific discovery. Source: <a href="https://www.nature.com/articles/s41524-022-00810-x">Li et al., 2022</a>. (<a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>)</figcaption> </figure> ] .references[ * Li et al. [Machine learning in concrete science: applications, challenges, and best practices](https://www.nature.com/articles/s41524-022-00810-x). Nature npj Computational Materials, 2022 ] ??? Concrete: However, these computational techniques require considerable computational resources and thus come with significant challenges, such as their limited time scales and the relatively small number of atoms in a simulated system. In addition, it may be difficult to validate these simulations with experiments, given the small time and length scales and high-fidelity measurements required. By leveraging existing datasets with data-driven models, ML can automatically learn implicit patterns and extract valuable information while accounting for the inherent complexity of concrete mixtures and their properties. --- ## Traditional scientific discovery loop .right-column-66[<br>.center[]] .left-column-33[ The .highlight1[traditional pipeline] for scientific discovery (paradigms 1-3): * relies on .highlight1[highly specialised human expertise], * it is .highlight1[time-consuming] and * .highlight1[financially and computationally expensive]. ] --- count: false ## Machine learning in the loop .right-column-66[<br>.center[]] .left-column-33[ A .highlight1[machine learning model] can be: * trained with data from _real-world_ experiments and * used to quickly and cheaply evaluate queries ] --- count: false ## Machine learning in the loop .right-column-66[<br>.center[]] .left-column-33[ A .highlight1[machine learning model] can be: * trained with data from _real-world_ experiments and * used to quickly and cheaply evaluate queries .conclusion[A machine learning model replacing real-world experiments can _only_ provide _linear_ gains.] .conclusion[Not enough if the search space is very large ($10^{180}$ stable materials)] ] --- count: false ## _Generative_ machine learning in the loop .context[Can we do better than _linear_?<br>An agent in the loop] .right-column-66[<br>.center[]] .left-column-33[ A .highlight1[machine learning **agent**] in the loop can: * .highlight1[learn structure] from the available data, * .highlight1[generalise] to unexplored regions of the search space and * .highlight1[build better queries] ] --- count: false ## _Generative_ machine learning in the loop .context[Can we do better than _linear_?<br>An agent in the loop] .right-column-66[<br>.center[]] .left-column-33[ A .highlight1[machine learning **agent**] in the loop can: * .highlight1[learn structure] from the available data, * .highlight1[generalise] to unexplored regions of the search space and * .highlight1[build better queries] .conclusion[A successful AL pipeline with an ML agent in the loop can provide _exponential_ gains.] ] --- count: false ## _Generative_ machine learning in the loop .context[GFlowNet as the agent.] .right-column-66[<br>.center[]] .left-column-33[ A .highlight1[machine learning **agent**] in the loop can: * .highlight1[learn structure] from the available data, * .highlight1[generalise] to unexplored regions of the search space and * .highlight1[build better queries] .conclusion[A successful AL pipeline with an ML agent in the loop can provide _exponential_ gains.] ] .references[ Jain et al.. [GFlowNets for AI-Driven Scientific Discovery](https://arxiv.org/abs/2302.00615). Digital Discovery, Royal Society of Chemistry, 2023. ] --- ## Machine learning for scientific discovery ### Challenges and limitations of existing methods .highlight1[Challenge]: very large search spaces. -- → Need for .highlight2[efficient search and generalisation] of underlying structure. -- .highlight1[Challenge]: underspecification of objective functions or metrics. -- → Need for .highlight2[diverse] candidates. -- .highlight1[Limitation]: Reinforcement learning and MCMC methods are good at optimisation but bad at mode mixing. -- → Need for .highlight2[multi-modal optimisation]. --- ## An intuitive toy task .context[Scientific discovery involves exploring in large, multi-modal search spaces.] <br> Task: find arrangements of Tetris pieces on the board that minimise the empty space. .left-column[ .center[] ] .right-column[    ] -- .conclusion[This task resembles designing DNA sequences or molecules or materials via fragments, to optimise certain properties.] --- ## An intuitive toy task .context[Scientific discovery involves exploring in large, multi-modal search spaces.] <br> Task: find arrangements of Tetris pieces on the board that minimise the empty space. .center[ <div style="display: flex"> <div style="flex: 25%;"> <figure> <img src="../assets/images/slides/tetris/board_empty.png" alt="Score 0/12" style="width: 30%"> <figcaption>Score: 0/12</figcaption> </figure> </div> <div style="flex: 25%;"> <figure> <img src="../assets/images/slides/tetris/board_score_4.png" alt="Score 4/12" style="width: 30%"> <figcaption>Score: 4/12</figcaption> </figure> </div> <div style="flex: 25%;"> <figure> <img src="../assets/images/slides/tetris/board_score_8.png" alt="Score 8/12" style="width: 30%"> <figcaption>Score: 8/12</figcaption> </figure> </div> <div style="flex: 25%;"> <figure> <img src="../assets/images/slides/tetris/board_score_12.png" alt="Score 12/12" style="width: 30%"> <figcaption>Score: 12/12</figcaption> </figure> </div> </div> ] --- ## An intuitive toy task .context[Scientific discovery involves exploring in large, multi-modal search spaces.] <br> Task: find arrangements of Tetris pieces on the board that minimise the empty space. .center[ <div style="display: flex"> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode1.png" alt="Score 0/12" style="width: 30%"> <figcaption>Score: 12/12</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode2.png" alt="Score 4/12" style="width: 30%"> <figcaption>Score: 12/12</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode3.png" alt="Score 8/12" style="width: 30%"> <figcaption>Score: 12/12</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode4.png" alt="Score 12/12" style="width: 30%"> <figcaption>Score: 12/12</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode5.png" alt="Score 12/12" style="width: 30%"> <figcaption>Score: 12/12</figcaption> </figure> </div> </div> ] .conclusion[The _reward function_ of this task has multiple modes. With a larger board and more pieces, the number of combinations and modes grow exponentially and the task of efficiently finding them is non-trivial for machine learning models.] --- name: title class: title, middle ## A gentle introduction to GFlowNets ### Part 2 .center[] --- ## GFlowNet in a nutshell .context[Existing ML methods struggle with large, multi-modal search spaces.] <br> Given a reward or objective function $R(x)$, GFlowNet can be seen a generative model trained to sample objects $x \in \cal X$ according to .highlight1[a sampling policy $\pi(x)$ proportional to the reward $R(x)$]: $\pi(x) \propto R(x)$ → Sampling proportionally to the reward function induces .highlight1[multi-modal search and diversity]. -- .left-column[ The policy $\pi_{\theta}(x)$ is modelled by a deep neural network, parameterised by $\theta$, thus providing .highlight1[amortised inference]. → Amortised inference can be thought of as _exploration with memory_, which induces .highlight1[systematic generalisation]. ] -- .right-column[ .center[] ] --- ## GFlowNet in a nutshell * Objects $x \in \cal X$ are constructed through a sequence of steps $\tau$ from an action space $\cal A$. * At each step of the trajectory $\tau=(s_0\rightarrow s_1 \rightarrow \dots \rightarrow s_f)$, we get a partially constructed object $s$ in state space $\cal S$. * This induces a directed acyclic graph (DAG) $\mathcal{G}=(\mathcal{S},\mathcal{A})$, with all possible constructions in the domain. .left-column[.center[]] .right-column[.center[ <div style="display: flex"> <div style="flex: 30%;"> <figure> <img src="../assets/images/slides/tetris/s0.png" alt="s0" style="width: 100%"> <figcaption>$s_0$</figcaption> </figure> </div> <div style="flex: 0.5%; padding: 40pt 0"> $\rightarrow$ </div> <div style="flex: 30%;"> <figure> <img src="../assets/images/slides/tetris/s1.png" alt="s1" style="width: 100%"> <figcaption>$s_1$</figcaption> </figure> </div> <div style="flex: 0.5%; padding: 40pt 0"> $\rightarrow$ </div> <div style="flex: 30%;"> <figure> <img src="../assets/images/slides/tetris/s2.png" alt="s2" style="width: 100%"> <figcaption>$s_2$</figcaption> </figure> </div> </div> ]] --- ## GFlowNet in a nutshell .context[GFlowNet induces a DAG $\mathcal{G}=(\mathcal{S},\mathcal{A})$, with all possible constructions in the domain.] <br> .left-column[.center[ <figure> <img src="../assets/images/slides/tetris/state_space.png" alt="State space" style="width: 80%"> <figcaption>State space $\cal S$</figcaption> </figure> ]] .right-column[.center[ <figure> <img src="../assets/images/slides/tetris/action_space.png" alt="Action space" style="width: 60%"> <figcaption>Action space $\cal A$</figcaption> </figure> ]] .conclusion[This terminology is reminiscent of reinforcement learning.] --- ## GFlowNet flows .context[The edges or transitions in the DAG can be quantified by their _flow_.] * Analogous to water-flow in pipes. * Trajectory Flow $F(\tau)$ denotes probability mass assigned to trajectory $\tau$. * State Flow $F(s)$ is the flow of all trajectories passing through the state $s$. * Edge Flow $F(s\rightarrow s')$ is the flow through a particular edge $s\rightarrow s'$. * Forward Policy $P_F$: $P\_F(s'|s) = \frac{F(s\rightarrow s')}{F(s)}$ * Backward Policy $P_B$: $P\_B(s|s') = \frac{F(s\rightarrow s')}{F(s')}$ .center[] .references[ Bengio et al. [Flow network based generative models for non-iterative diverse candidate generation](https://arxiv.org/abs/2106.04399), NeurIPS, 2021. ] ??? Not to be confused with normalizing flows! --- ## Principle of conservation as a training objective .right-column-33[.center[]] .left-column-66[ **Consistent Flow**: Flow $F$ satisfies the _flow consistency equation_ $$\sum\_{s' \in \text{Parent}(s)} F\_\theta(s' \rightarrow s) = \sum\_{s'' \in \text{Child}(s)} F\_\theta(s \rightarrow s')$$ **Theorem**: For a consistent flow $F$ with terminal flow set as the reward $F(x\rightarrow s_f)=R(x)$, the forward policy samples $x$ proportionally to $R(x)$. $$\pi(x)\propto R(x)$$ **Corollary**: The flow at $s_0$, $F(s_0)$ is the partition function $Z$! ] .references[ Bengio et al. [Flow network based generative models for non-iterative diverse candidate generation](https://arxiv.org/abs/2106.04399), NeurIPS, 2021. ] --- ## Principle of conservation as a training objective <p> $$\sum\_{s' \in \text{Parent}(s)} F\_\theta(s' \rightarrow s) = \sum\_{s'' \in \text{Child}(s)} F\_\theta(s \rightarrow s')$$ </p> * **Flow Matching Objective**: $$\mathcal{L}\_{FM}(s; \theta) = \left(\log \frac{\sum\_{s'\in \text{Parent}(s)} F\_\theta(s'{\rightarrow} s)}{\sum\_{s'' \in \text{Child}(s)}F\_\theta(s{\rightarrow} s'')}\right)^2$$ * **Trajectory Balance**: $$\mathcal{L}\_{TB} (\tau;\theta) = \left(\log \frac{Z\_\theta \prod\_{s{\rightarrow} s' \in \tau}P\_{F\_\theta}(s'|s)}{R(x)\prod\_{s\rightarrow s' \in \tau} P\_{B\_\theta}(s|s') }\right)^2$$ --- ## Results ### Tetris GFlowNets .context[If the model is sufficiently trained, the sampling policy $\pi(x)$ should be proportional to the reward $R(x)$: $\pi(x) \propto R(x)$] <br> .center[ <div style="display: flex"> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode1.png" alt="Score 0/12" style="width: 30%"> <figcaption>$\pi(x) = 8.12~\%$</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode2.png" alt="Score 4/12" style="width: 30%"> <figcaption>$\pi(x) = 8.96~\%$</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode3.png" alt="Score 8/12" style="width: 30%"> <figcaption>$\pi(x) = 8.61~\%$</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode4.png" alt="Score 12/12" style="width: 30%"> <figcaption>$\pi(x) = 9.16~\%$</figcaption> </figure> </div> <div style="flex: 20%;"> <figure> <img src="../assets/images/slides/tetris/mode5.png" alt="Score 12/12" style="width: 30%"> <figcaption>$\pi(x) = 8.39~\%$</figcaption> </figure> </div> </div> ] After training, GFlowNet samples a mode with probability 43.24 %. .footnote[The energy function $\varepsilon(x)$ is the fraction of the board occupied by pieces and the reward function is $R(X) = \varepsilon(x)^4$ to disproportionally favour the discovery of modes.] --- ## Results ### Hyper-grid and molecule fragments .context[GFlowNet has been successfully trained in other toy and practically relevant tasks. .cite[(Bengio et al., 2019)]] .columns-3-left[ .highlight1[Hyper-grid]: The action space is in which dimension to move and the reward function has high reward in the corners. .center[  ] ] -- .columns-3-center[ .highlight1[Small molecules]: The action space is molecular fragments and the reward function is the binding energy to a particular protein. .center[  ] ] -- .columns-3-right[ .highlight1[Active learning with molecules]: Multi-round active learning with a limited oracle budget. .center[  ] ] -- .conclusion[GFlowNet is able to efficiently explore the search space and generalise to unseen modes of the reward.] --- ## GFlowNet extensions ### Multi-objective GFlowNets We have extended GFlowNets to handle multi-objective optimisation and not only cover the Pareto front but also sample diverse objects at each pointin the Pareto front. .center[  ] .references[ Jain et al. [Multi-Objective GFlowNets](https://arxiv.org/abs/2210.12765), arXiv 2210.12765, 2022. ] --- ## GFlowNet extensions ### Continuous GFlowNets We have recently generalised the theory and implementation of GFlowNets to encompass both discrete and continuous or hybrid state spaces. .center[  ] .references[ Lahlou et al. [A Theory of Continuous Generative Flow Networks](https://arxiv.org/abs/2301.12594), arXiv 2301.12594, 2023. ] --- name: title class: title, middle ## Application: Biological Sequence Design ### Part 3a .center[] --- ## Biological sequence design: anti-microbial peptides * Peptides are short chains of amino acids (proteins) .cite[(Pirtskhalava et al., 2021)] * The goal is to find peptides with anti-microbial properties * We consider chains of length 50 or shorter, with a vocabulary of 20 aminoacids ($>10^{65}$) * Data set: 6438 positive (anti-microbial samples) and 9522 negative samples. .references[ * Pirtskhalava et al. DBAASP V3: Database of antimicrobial/cytotoxic activity and structure of peptides as a resource for development of new therapeutics. Nucleic Acids Research, 2021. * Jain et al. [Biological Sequence Design with GFlowNets](https://arxiv.org/abs/2203.04115), ICML, 2022. ] ??? 3219 AMP, 4611 non-AMP (anti-microbial properties) --- ## Application ### Biological sequence design: anti-microbial peptides .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] --- count: false ## Application ### Biological sequence design: anti-microbial peptides .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **Oracle**: MLPs trained on a separate partition of the data. In other tasks, such as DNA aptamers, we have access to computational libraries. ] --- ## Application ### Biological sequence design: anti-microbial peptides .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **Model**: MLPs and transformers. ] --- count: false ## Application ### Biological sequence design: anti-microbial peptides .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **Agent**: GFlowNet .center[] ] --- ## Methodology ### Algorithm .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **1**: Generate a _small_ initial _labelled_ data set from the oracle: $$\mathcal{D} = \mathcal{D_0}$$ In our main experiments: * $|\mathcal{D_0}| = 7830$ * Length: up to 50 * Alphabet: 20 aminoacids ] --- count: false ## Methodology ### Algorithm .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **2**: Train model by minimising the error on $\mathcal{D}$: $$\min L(f_{\theta}(\mathcal{D}))$$ In our main experiments we train with MC dropout or an ensemble of MLPs that provides with both _energy_ and _uncertainty_. ] --- count: false ## Methodology ### Algorithm .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **3**: Train GFlowNet until convergence using the ML model as a proxy oracle: .center[] ] --- count: false ## Methodology ### Algorithm .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **4**: Generate samples with GFlowNet and select a query of best candidates We select 1000 samples per iteration. ] --- count: false ## Methodology ### Algorithm .context[An **active learning** pipeline for biological sequence design] .right-column-66[.center[]] .left-column-33[ **5**: Score the selected query with the oracle ($\mathcal{D_1}$) and add to the data set: $$\mathcal{D} = \mathcal{D} \cup \mathcal{D_1}$$ Then repeat all steps for $k=10$ iterations. ] --- ## Evaluation ### Desiderata for candidates .context[We look for a .highlight1[diverse] set of .highlight1[good] candidates.] The set of top $k$ candidates: $\mathcal{D}_{Best} = TopK(\mathcal{D}_K \backslash \mathcal{D}_0)$ * .highlight1[Performance / usefulness score]: mean score of $\mathcal{D}_{Best}$ * .highlight1[Diversity]: mean distance within $\mathcal{D}_{Best}$ * .highlight1[Novelty]: mean distance with between $\mathcal{D}_{Best}$ and $\mathcal{D}_0$ --     .conclusion[A set of candidates should be evaluated holistically, considering all three metrics.] --- ## Evaluation ### Baselines Three representative recent machine learning models for sequence design: * DynaPPO: Active Learning with RL as Generator .cite[Angermueller et al., 2019] * AmortizedBO: Bayesian Optimization with RL-based Genetic Algorithm .cite[Swersky et al., 2020] * COMs: Deep Model Based Optimization .cite[Trabucco et al., 2021] .references[ * Angermueller et al. Model-based reinforcement learning for biological sequence design. ICLR, 2019. * Swersky et al. Amortized bayesian optimization over discrete spaces. PMLR, 2020. * Trabucco et al. Conservative objective models for effective offline model-based optimization. ICML, 2021. ] --- ## Results .context[Active learning with 10 rounds, $b = 1000$, $K = 100$.] * GFlowNet generates sequences with .highlight1[score] on par or higher than the strongest baseline. * GFlowNet generates much more .highlight1[diverse] and .highlight1[novel] samples than the baselines. .center[ <figure> <img src="../assets/images/slides/gfn-seq-design/table1.png" alt="Metrics with $K=100$" style="width: 85%"> <figcaption>Metrics with $K=100$</figcaption> </figure> ] -- .smaller[ * AmortizedBO generated nonsensical peptides because it is designed for fixed-length sequences. * Methods such as COMs, which perform local search around known candidates, perform poorly when the goal is to generate .highlight1[large], .highlight1[diverse] and .highlight1[novel] batches. ] -- .conclusion[GFlowNet succeeds at generating sequences that satisfy all three metrics: high score, diversity and novelty.] --- name: title class: title, middle ## Application: Generation of Crystal Structures ### Part 3b .center[] --- ## Generation of Crystal Structures ### _Work in progress..._ We are working on using GFlowNets to explore the space of crystals and sample new, diverse, stable materials that optimise certain properties. * State space: stoichiometry, crystal system, point symmetry, space group, Wyckoff positions, etc. * Action space: the constrained selection of crystal structure parameters and properties. * Advantage of GFlowNets: flexibility to incorporate domain knowledge in the generation structure. -- ### Applications * Electrocatalyst design * Solid-state ionic super conductors * ... --- name: title class: title, middle ## Summary and conclusions .center[] --- ## Summary and conclusions * Tackling the most pressing problems for humanity, such as the climate crisis and the threat of global pandemics, requires .highlight1[accelerating the pace of scientific discovery]. * While in the .highlight1[data-drive science paradigm], our current AI tools are not enough to truly utilize all the information and resources at our disposal. * AI-driven scientific discovery demands learning methods that can .highlight1[efficiently discover diverse candidates in very large, multi-modal search spaces]. * .highlight1[GFlowNet] is a learning method for amortised inference that can sample proportionally to a reward function. * GFlowNet has been proven successful or is promising in practically relevant and challenging tasks, such as .highlight1[biological sequence design], .highlight1[molecular conformation], .highlight1[crystal structure generation], etc. .references[ Jain et al.. [GFlowNets for AI-Driven Scientific Discovery](https://arxiv.org/abs/2302.00615). Digital Discovery, Royal Society of Chemistry, 2023. ] -- .highlight2[Open source code]: [github.com/alexhernandezgarcia/gflownet](https://github.com/alexhernandezgarcia/gflownet) --- name: title class: title, middle ## Thanks! Questions?  Alex Hernández-García (he/il/él) .center[ <a href="https://mila.quebec/"><img src="../assets/images/slides/logos/mila-beige.png" alt="Mila" style="height: 3em"></a>      <a href="https://www.umontreal.ca/"><img src="../assets/images/slides/logos/udem-white.png" alt="UdeM" style="height: 3em"></a> ] .footer[[alexhernandezgarcia.github.io](https://alexhernandezgarcia.github.io/) | [alex.hernandez-garcia@mila.quebec](mailto:alex.hernandez-garcia@mila.quebec)]<br> .footer[[@alexhg@scholar.social](https://scholar.social/@alexhg) [](https://scholar.social/@alexhg) | [@alexhdezgcia](https://twitter.com/alexhdezgcia) [](https://twitter.com/alexhdezgcia)] .smaller[.footer[ Slides: [alexhernandezgarcia.github.io/slides/mmiccs-mar23](https://alexhernandezgarcia.github.io/slides/mmiccs-mar23) ]] <