Testing out RAG (retrieval augmentation generation) for Artificial Intelligence

[AlexB23]

Hello folks. Sometime tomorrow, I will be testing out a local AI on my computer that can retrieve information from documents on my laptop.

Retrieval Augmentation Generation (RAG) is where an AI can use documents as a source for output. After plugging in the Holy Bible, a Catholic Catechism and a book about 2000 years of Church history, I am hoping that this new software can improve my devotionals. If not, it can just work as a regular AI, if I turn the RAG mode off.

RAG info: What is retrieval-augmented generation (RAG)?

My computer is using GPT4ALL v3.00, along with Mistral 7B v0.3 (a step up from v0.2). All this stuff runs offline, and does not need an internet connection.

 

[AlexB23]

Alright, so the system works, as it now references a Catechism for the Catholics. Now, I am adding medical textbooks to the AI, to see if it knows basic medical science. Maybe later on, I could add some Orthodox and Protestant documents into the system. Welcome to the future guys, where artificial intelligence could be run offline on an airplane or in a forest. Imagine a biologist in a remote area asking the AI for a description of a rare species of plant* or animal, and the AI tells her what it is based upon an image uploaded to the AI. The AI on my computer is not capable of image recognition, but other AI could be.


*Plant identification tool (offline): Pl@ntNet offline/embedded: identify plants anywhere without connection - Pl@ntNet


Catechism PDF added to AI.

 

[elytron]

Right now am sort of wary of installing an AI on my main computer. Maybe if I had a spare desktop I would try and experiment.

In the meantime I will just stick with the online ones which don't have access to my files.

 

[AlexB23]

Right now am sort of wary of installing an AI on my main computer. Maybe if I had a spare desktop I would try and experiment.

In the meantime I will just stick with the online ones which don't have access to my files.

Thar is understandable. For me, I am the opposite, and am wary of any online AI, as online AI may collect telemetry. What type of desktop do you have? I have a Microsoft Surface Book 2 which my parents got me for college which I used in 2018 and 2019. It still works well in 2024. For myself, I can prevent the AI from having access to the files, cos the RAG mode can be turned off, or not used.

This is the AI explaining the Non Expedit, some confusing Catholic thing, as I am Catholic myself and someone asked a question about it in the Catholic section of the forum.

 

[AlexB23]

Just installed appliance instruction manuals onto the AI, so if anything goes wrong, I can troubleshoot my appliances in the future. Luckily, most of my appliances are new, so there will be no issues for a while.

 

[The Liturgist]

Thar is understandable. For me, I am the opposite, and am wary of any online AI, as online AI may collect telemetry. What type of desktop do you have? I have a Microsoft Surface Book 2 which my parents got me for college which I used in 2018 and 2019. It still works well in 2024. For myself, I can prevent the AI from having access to the files, cos the RAG mode can be turned off, or not used.

This is the AI explaining the Non Expedit, some confusing Catholic thing, as I am Catholic myself and someone asked a question about it in the Catholic section of the forum.

View attachment 351451

I have some vintage IBM and Sun Microsystems workstations with a mixture of POWER (like PowerPC Macs, but more powerful, with more instructions) or UltraSPARC RISC and Intel x64 CPUs and some newer workstations and laptops mainly from Lenovo, with a single x64 iMac from 2018, and a few older Dell machines as well, and one HP laptop that does a very good job with cooling.

 

[AlexB23]

I have some vintage IBM and Sun Microsystems workstations with a mixture of POWER (like PowerPC Macs, but more powerful, with more instructions) or UltraSPARC RISC and Intel x64 CPUs and some newer workstations and laptops mainly from Lenovo, with a single x64 iMac from 2018, and a few older Dell machines as well, and one HP laptop that does a very good job with cooling.

Those vintage systems may struggle with large AI models. Your newer computers might run AI better. I recommend any computer made in the past 5 years, and more than 16 GB of RAM for smaller AI models, and larger RAM amounts for larger AI models. You would need a small mainframe to run Llama 3.1 405B.

 

[AlexB23]

Update: It seems that Retrieval Augmentation Generation (RAG) has been a success. The AI struggles with RAG when it comes to political platform documents for some reason, as the AI thinks that the year 2024 is still in the future, but does much better for religious documents (theology) and sports rule books.

 

[The Liturgist]

Those vintage systems may struggle with large AI models. Your newer computers might run AI better. I recommend any computer made in the past 5 years, and more than 16 GB of RAM for smaller AI models, and larger RAM amounts for larger AI models. You would need a small mainframe to run Llama 3.1 405B.

Don’t underestimate the power of my RISC systems - they are massively multithreaded, which is good for AI operations. The same is true of the IBM Power Cell CPU. And the upshot is not having to pound a GPU.

Now that being said, admittedly, a modern CPU with AI optimized cores could eat their lunch, which is why I’m planning either a new workstation with dedicated AI cores or refurbishing some newish rackmount servers for this purpose.

I prefer servers that have remote management, whether they are rackmount or tower, since I really don’t want to get into the OS on an internals respect. I would probably use late model Cisco UCS servers or HP Enterprise or Lenovo.

 

[AlexB23]

Don’t underestimate the power of my RISC systems - they are massively multithreaded, which is good for AI operations. The same is true of the IBM Power Cell CPU. And the upshot is not having to pound a GPU.

Now that being said, admittedly, a modern CPU with AI optimized cores could eat their lunch, which is why I’m planning either a new workstation with dedicated AI cores or refurbishing some newish rackmount servers for this purpose.

I prefer servers that have remote management, whether they are rackmount or tower, since I really don’t want to get into the OS on an internals respect. I would probably use late model Cisco UCS servers or HP Enterprise or Lenovo.

Interesting stuff. RISC-V is the newest RISC system, developed in the 21st century.