anjanesh Thank you for asking. I will do my best to explain a bit of possible solutions and why I feel this kind of approach is amazing and the future of the industry for big challenges like this.
How does one feed very large input data sets?
AI has two kinds of compute problems in general. Training and inference. Training creates the blobs which are the 'model' effectively a kind of noise for a neural network's storage to function how it does. Forgive me as I am not a mathematician and can only vaguely ponder into the orb that is the inner-workings of the math. But for the sake of logistics we have two. Training and inference.
The current hardware models have solutions to radically improve inference on trained models by burning their own ASICS. That will come in short time. However there is no foreseeable way to increase efficiency on Training except the same kind of hardware as GPU compute. Of course Nvidia and other competitors which might arise (they hold the monopoly for now) have bigger and better GPU compute. But then the issue of privacy arises, as well as cost, and the existing eco system of legacy hardware which impacts ecology concerns.
For your question we would be focusing on Training.
Under the current and foreseeable cloud compute model you would do this like you do any other large dataset, think minio or mongodb. Once the data is in the DB you point pytorch at the data and let the GPU rip.
The logistic concerns I mentioned previous would then take effect on the practical calculation and execution of said data.
Personally, if I were to do this myself right now. The process would look something like,
- Install MongoDB or Minio on a server/cluster on my home lab
- Put the data there
- Find the model I want off of hugging face
- Have GPT4 help me turbo some code boiler plate for pytorch to connect all this together
- Let it rip, tuning or adding legacy GPU from ebay as required.
This is going to be a stop-gap strategy until the industry distributes the next gen compute modules, however those dice might roll.
How would Opalstack be useful for this? We want to provide the web2 layer, be available as support for any developer who wants to build these projects, and be on the edge of emerging technology as it pertains to web2 developers. I don't see the web2 component going anywhere, its like text to the Gutenberg press, they didn't get rid of the letter A, web2 is the alphabet. and this new printing press will require it to be the UX.
P.S. I should note you can train a model from scratch or tune an existing model, and for practical application we are in the age of tuning models released by the giants, because the economy of time/scale makes it neigh impossible to catch up to them organically, not impossible, but for our application as web developers that is the reality. Each model has its own software license as well, and this is a fun layer to all of this decision making while engineering solutions.
Hope this is a useful answer. Please reach out if you have any other questions.
