isomorphic: mapping between 2 structures that preserver the structure and can be reversed.

” as organisms become more and more complex through evolution, they need to model reality with increasing accuracy to stay fit. At all times, their representation of reality must be homomorphic with reality itself. Or in other words, the true structure of our world must be preserved when converted into your brain’s representation of it.”

from here https://savsidorov.substack.com/p/tldr-the-interface-theory-of-perception

Homomorphism in Algebra is a structure-preserivng map between 2 algebric structures of the same type (not reversible like isomorphic?)

Joscha Bach on GPT-3 https://www.youtube.com/watch?v=FMfA6i60WDA GPT-3 moves in a semantic space, masters relations between words, but can only deepfake understading

Joshua built the MicroPSI architecture based on PSI Theory https://en.wikipedia.org/wiki/Psi-theory#MicroPsi_architecture

Curious Wavefunction makes a history of information and thermodynamics http://wavefunction.fieldofscience.com/2020/07/brains-computation-and-thermodynamics.html

ends it with the idea that our brain is a mixture of digital and analog processes, as posited by Von Neumann

Sav Sidorov another Joscha Bach video with highlights https://savsidorov.substack.com/p/tldr-joscha-bach-artificial-consciousness

“Some people think that a simulation can’t be conscious, and only a physical system can, but they got it completely backwards. A physical system cannot be conscious, only a simulation could be conscious.”

like Communism was Eletrification + power of the soviet.

The bitter lesson o fRich Sutton http://www.incompleteideas.net/IncIdeas/BitterLesson.html

from Gwern May newsletter on scaling and metalearning:

“The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is ‘just’ simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale.”

This is related somehow, distributed intelligence and fungi from The Curious Wavefunction, Life Distributed http://wavefunction.fieldofscience.com/2020/09/life-distributed.html

start from Gwern here https://www.gwern.net/newsletter/2020/05#gpt-3

parameters scaling in GPT-3 does not run into linear scaling of performance nor dimishing returns. Rather it shows metalearning enhancing the performance

It was forecast by Moraves and since we are in a fat tail phenomenon this holds true: “the scaling hypothesis is so unpopular an idea, and difficult to prove in advance rather than as a fait accompli“. Before GPT-3 another epiphany on the scaling was the google cat moment which started the deep learning craze

Another idea which I like is that models like GPT-3 are definitely cheap and if they show superlinear growth it is a no brainer to go for bigger and more complex models, it is along way before matching the billions of expenses for Cern or nuclear fusion.

Carig Venter synthetic bacteria project cost us 40 milion, ground braking orojects costing so little should not be foregone

BTW to grasp the idea of how there could be a scaling benefit in growing deep learning sizes, go no further that a simple, unfounded but suggestive analogy with Metcalfe law of networks, network value grows with the square of nodes.