[vip-help] Stuck on code with unknown reason for collapse

[Roytman, Avigdor]

Hi,

For our TransGAN code, I decided to try and simplify in order to find the error. I rewrote the Generator and Discriminator for the MNIST dataset where the images are a lot smaller and only using the TransformerBlock without GridSelfAttention or any extra stuff to try and get something working as simple as possible. This ended up producing results like shown below during the training where the top are the real photos, and the bottom are the generated photos, and it collapses to this almost instantly:
  [cid:e9582ff5-b232-4d0e-8b5c-1bbe3da2b341]
I have quadruple-checked all the sizes of every part of the code, and everything seems to be set up as it should be. I used to have assert statements after every line of code and they all did not throw any errors. I have tried playing around with the learning rate, weight initialization, dropout rate, embed dimension size, feed-forward dimension size, and every hyperparameter I can change. Nothing seems to work, and I don't know how to get simpler than the MNIST dataset. I'm completely stuck and don't know what to do from here to try and get it working. All this code is in the Train-MNIST.py and TransformerBlock.py files in the TransGAN folder. The other files in the folder are for the CelebA dataset and aren't being used for this to try and keep simplicity. Is there any way you can help?

Thank you,
Avi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ECN/mailman/archives/vip-help/attachments/20240404/22034d3a/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 48236 bytes
Desc: image.png
URL: </ECN/mailman/archives/vip-help/attachments/20240404/22034d3a/attachment-0001.png>