Description

Hi all,


A couple years ago in this post (https://www.reddit.com/r/UFOs/comments/1ah3hat/scanning_project_blue_… outlined a project to use Large Language Models (LLMs) like GPT-4 for digitizing poorly scanned old case files from Project Sign, Grudge, and Bluebook to make them easily searchable for the community.


Today we have achieved that goal and would like to share the result and a call for action!

Behold:

theflyingsaucerfiles.com (http://theflyingsaucerfiles.com)

\- a UFO history project!

the landing page 🛸 (https://preview.redd.it/j5it3798wf5h1.png?width=1344&format=png&auto=…)

Features:

\- This is a searchable database of 10,807 historical UFO casefiles from projects sign, grudge and bluebook

\- Users can search by year, conclusion, number of witnesses, photographs included

\- There is also a map of cases! you can browse your home state/town/country for historical UFO cases that were reported to and investigated by the USAF

\- Cases with photos are also labeled but warning! many of them do not actually contain photos. Hopefully we can improve the quality of that metric.


Best practices and calls to action:

\- The text is, of course, scanned and digitized by AI models. It is not always reliable for details like including photographs or exact number of witnesses, but the data quality is the best we could get it so far

\- Honestly a lot of cases still need big improvements on the text, but it is hard to target them programatically.

\- Please use the best practice of navigating to the original PDF on the page to confirm details

\- If you click the text while reading a case, you will get routed to the original PDF on the exact page you were reading

\- If you spot problems in the text ⚠️ we kindly ask the community:

🙏 please provide feedback using the form! 🙏

The data can then be targeted for improvement!


\- If you have ideas for more ways to search this data, please comment on this post :) I am going to take a break on this for awhile, as it's been quite a lot of work, but your ideas are the key to making this website the best it can be.


Another interesting case we found and an example of using the site:

When looking for large witness counts, I came across this file

summary details at the top (https://preview.redd.it/o2cx9t5e2g5h1.png?width=992&format=png&auto=w…)

ai generated text snippet (https://preview.redd.it/wg6lsp0h2g5h1.png?width=646&format=png&auto=w…)

original pdf snippet (https://preview.redd.it/kzjar85j2g5h1.png?width=918&format=png&auto=w…)

You can see the text isn't perfect if you compare to the original, and it hallucinated a lot of text.

BUT the technique made it searchable! We hope to improve the data quality incrementally over time still. Problems like this are due to usage of smaller cheaper models.

Has anyone heard of this case? I certainly hadn't!

Other info:


The website is open source, if you have better ideas or want to build on this, have at it!

https://github.com/dansterdam/blue\_book\_searcher (https://github.com/dansterdam/blue_book_searcher)

Recap:

The outcomes from the original post:

* Project Blue Book's 10,000+ case files (130k+ pages) scanned into text format ✅
* Periodic updates on the process and uploads of newly scanned data ✅
* Data uploaded to our github in an organized format for ease of download and searchability ✅
* Nice to have: data indexed to a database and made available on a website for ease of browsing and searching. ✅

So, have at it! and read some interesting cases!!


Thank you:

Thanks to r/UFOs and everyone here for the lovely comments on the other post and especially the donations that helped us get to the finish line after all this time!

Thanks to friends and family that supported this crazed idea and didn't let me give up!