Add latex & text recognition to screenshots

[Zabooby]

i just saw the recent nightly blog and having latex text recognition for screenshots would be super useful for university students like me. I often find myself needing to take screenshots from books/ pdfs online that do not allow text selection and then using a random ocr website to extract text/formulae. It is super slow, clunky and probably bad for privacy and security. Having integration from within Firefox would be amazing for a lot of students. There are a lot of projects on GitHub with this functionality, hope this can help.

[Jon]

Thanks for submitting an idea to the Mozilla Connect community! Your idea is now open to votes (aka kudos) and comments.

[sfoster]

Thanks for posting this. It feeds into an ongoing discussion we've been having about screenshots and the use case of capturing and archiving content viewed in the browser, where an image format wouldn't be the first choice given other options.

The other part here is an in-browser way to apply OCR to any image - including screenshots. We do have text-extraction as a feature, on MacOS only for the moment. The tentative plan is to bring this feature to the other OSes in time. That path makes sense if you are working with an existing image and have no ability to interact with the source content. But for your use case it seems like a fallback, worst-case scenario. Capturing text as an image loses a lot of context, accessibility and semantic hints which might be present.

Finally, the latex recognition: that is a perennial issue. In fact, if memory serves, the ability to represent formulae and similar constructs was one of several original motivations for adding images to HTML. So we've gone full circle. Realistically, I think our best path forward on this would be to improve the ability for extensions to add functionality when working with images - again, including screenshots.

[Zabooby]

@sfosterI'm on windows 11 rn and look forward to using this feature. A lot of online textbooks don't allow text selection at all and it would be great to have this feature.

I wouldn't mind at all if it latex recognition was implemented as an extension.

[orangecandycorn]

I agree that OCR from screenshots in Firefox would be useful for random documents that do not have text characters already. +1 as a fellow student!