Socrates AI
Interact with a from-scratch trained Transformer model, and gain wisdom.
Check out the current progress here: progress
Note: This model is still in development, responses are limited. I'd recommend using it as a sentence completion tool in a philosophical style, for example try entering: "The meaning of virtue is"
About this model:
Socrates AI is a very small transformer language model (~7 million parameters, 4 layers each with 4 attention heads) built entirely from scratch, including its own byte level BPE tokenizer. Nothing here is pretrained or fine tuned from an existing model.
The training dataset consists of 634 public domain philosophy books from Project Gutenberg's Philosophy bookshelves. This ended up being around 256 million characters, which is about 64 million tokens after cleaning.
Model responses are still pretty limited at the moment, which can be quickly found out after trying some different prompts. But the good news is there is definately a 'philosophical' style to the answers, and I plan to train it more to try and get improved answers. It's currently at step 9000 in training, and the plan is to get it to 18,000 and then re-evaluate.