Rambling about performing "surgery" on neural networks and other random thoughts hopefully related
Fruit Fly paper mapping visual systems
https://www.biorxiv.org/content/10.1101/2023.03.11.532232v1
Paper defeating professionals in Games
https://arxiv.org/abs/1912.06680
https://www.researchgate.net/publication/337967467_Neural_Network_Surgery_with_Sets
So as we deep dive into the fruit fly paper, I'm revisiting some gaming papers that OpenAI applied "surgery" toward neural networks years ago (2019). They used sets and gradients to detect which parts do and dont need retraining (likely an older than 2019 technique at that time). I'm wondering what the SoTA is now for transferring trained weights from one network to another. So if we're trained on the visual system, how would we even begin to transfer those weights toward its head movement or food drive? How would we even label these things?
When it comes to the work at Allen institute, yall said they're killing mice and scanning the brain during activity like eating or drinking? So they're labeling those mouse brain activities. At future stages, we would end up "animating" a 3 second action of a mouse brain drinking water? So we go from image generation to video generation same as how we start from simulating one frame of activity in the brain to simulating many frames?
So if we start with maybe fruit fly data, we label the scan's functionality while its trying to eat or trying to move.
Scan a human drinking water and transfer the weights of the mouse drinking water over to the human?
But if "biological representations" are within some hidden layers, then aren't we still stuck not knowing what's happening biologically still?
Is below an example of a "lofi" equivalent fruit fly paper's technique?
https://sites.google.com/view/stablediffusion-with-brain/
Maybe this is why "lofi" techniques are becoming more popular? It's because of BCI's? Maybe people betting more money toward short term goals in BCI's such as gaming and allow new interfaces for human paralysis?
Connectome-constrained deep mechanistic networks predict neural res...
We can now measure the connectivity of every neuron in a neural circuit, but we are still blind to other biological details, including the dynamical characteristics of each neuron. The degree to which connectivity measurements alone can inform understanding of neural computation is an open question. Here we show that with only measurements of th...
arXiv.org
Dota 2 with Large Scale Deep Reinforcement Learning
On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems....
Stable Diffusion with Brain Activity
Accepted at CVPR 2023
Yu Takagi* 1,2 , Shinji Nishimoto 1,2
1. Graduate School of Frontier Biosciences, Osaka University, Japan
2. CiNet, NICT, Japan
0 Replies