Share your feedback on the AI services experiment in Nightly

[asafko]

Hi folks, 

In the next few days, we will start the Nightly experiment which provides easy access to AI services from the sidebar. This functionality is entirely optional, and it’s there to see if it’s a helpful addition to Firefox. It is not built into any core functionality and needs to be turned on by you to see it. 

If you want to try the experiment, activate it via Nightly Settings > Nightly Experiments (please see full instructions here). 

We’d love to hear your feedback once you try out the feature, and we’re open to all your ideas and thoughts, whether it’s small tweaks to the current experience or big, creative suggestions that could boost your productivity and make accessing your favorite tools and services in Firefox even easier.

Thanks so much for helping us improve Firefox!

[DenJohn]

NO

[Mte90]

The announcement is not clear that is not started yet.

Also it isn't clear if it will require a personal token for this services and what settings are available.

[Mardak]

Try checking Nightly for updates. It should be available with 2024-06-25 builds with settings dropdown choices of 4 providers. No personal token is necessary as the chatbot will reuse your usual tab's cookies if any, e.g., ChatGPT doesn't require a login and can reuse your login or subscription for GPT-4o "omni."

 

If you're comfortable with changing about:config, you can set "browser.ml.chat.provider" to any URL to open it in the sidebar, and if that page accepts passing in prompts with ?q=, the context menu functionality will work too. You can create your own prompts by adding a new String pref, e.g., "browser.ml.chat.prompts.a" set to "Tell me more"

 

[Mte90]

I can confirm that works 🙂

[Mte90]

The first thing that I notice that there isn't a way to close the vertical window...

The only way I found is to press Customize Sidebar that has an X so I can close the AI sidebar.

[marianoguerra]

will this be exposed for js developers? chrome canary experiments with gemini seem to expose a window.ai object you can use to use the capabilities from code.

if so please try to talk to each other and standardize an MVP 🙂

[Mardak]

This current chatbot experiment does not make use of LLM / chat completion-type APIs, but there's a path to building a more integrated Firefox chatbot powered by local or remote inference APIs that could be exposed to developers both for the web and add-ons.

[magentapuppy]

yeah keep adding this useless and privacy invasive stuff that *all* the big corps are doing, rather than adding features that actually differentiate it, particularly as a less 'BS' browser option 🙄

[Mardak]

We included chatbot providers like HuggingChat which says: "We endorse Privacy by Design. As such, your conversations are private to you and will not be shared with anyone, including model authors, for any purpose, including for research or model training purposes."

Additionally, if you don't want to use a remotely hosted chatbot, you could use one that's on-device so data doesn't leave your computer while getting the benefits of optional AI.

[lackey]

AI has myriad ethical and environmental issues behind it. The first two options offered by Mozilla are created by some of the most invasive, unethical, environmentally destructive companies in the AI space. Google and Microsoft are not known for being private, not even a little bit.

The general issues behind AI, and the fact companies like HuFace need to scrape data without consent or permission, makes me question their promise to keep the data I offer them private.

These new changes appear antithetical to the Mozilla Manifesto.

I believe Mozilla should cease their attempts to integrate AI into the browser, and to reevaluate the ethical implications of promoting these products by these companies.

[magentapuppy]

Unfortunately I do not count scraping people's data without consent of any kind "private by design"

And wow, it must take an awful person to only consider the 'end user' as the one deserving of privacy. Like, just, wow.

[-0]

Why does this need to be shipped with Firefox? Can't this be an extension? I want less AI, not more. 😞

[Mardak]

This is currently a first integration of AI with plans for more optional functionality that don't require a sidebar interface, e.g., suggesting tabs that are relevant to your current activity or name for a tab group. Getting to more seamless integrations would likely be trickier as an extension and can be implemented with minimal impact for those who don't want it.

[duskord]

Can you also provide Claude?

[Mardak]

You can configure a custom provider to Anthropic Claude for easy access to chat while on another tab. Currently the context menu passing in a prompt doesn't work, but we'll work with them for better compatibility.

 

[TS25714]

This will absolutely need a second sidebar, the same way Edge already has (left side for tabs, right side for copilot etc, for example). Right now we would have to decide between Sidebery and similar (even the upcoming efforts to bring vertical tabs natively to FX) which is not worth it IMHO. Vertical tabs > shortcut to chatbot.

[Mardak]

These Nightly builds have the new sidebar experience with vertical tabs in development along with the chatbot experiment. We're still working on integrating these features, but you can have both active today.

 

[aminought]

Sidebery > native vertical tabs. Also:
1. Each sidebar extension should have their own width. It is very inconvenient to look at the result of the AI Chatbot in a small panel in which I usually have Sidebery.
2. Some sidebar extensions, like AI Chatbot, should be able to open in the overlay so that I can return to Sidebery without additional actions.
3. A setting is needed to select the language in which the chatbot should respond.

Thank you!

[Iron56716]

I honestly like the AI Chatbot integration. I'm hoping Google AI Studio will be added officially.

It'd be even better if the sidebar could get official web page integrations like Edge or Floorp, That feature is very handy.

[Mardak]

Curious, is there a particular functionality you like about Google AI Studio? The additional settings for Gemini model choice? Model tuning? Here's using it as a custom provider, but I had to manually copy/paste the prompt:

 

[Iron56716]

There's quite a lot of things I like about Google AI Studio. Firstly, the ability to completely turn off all the safety filters, it's much easier to chat with the AI without them. Secondly, like you mentioned - the Gemini Model choice. I can change the model to any different one right in the conversation without the need to open a new chat. And of course, there's some extra perks like Custom Instructions, Model tuning, etc. Last but not least, AI Studio allows the use of Gemini 1.5 Pro without the subscription (unlike the regular Gemini websites. Although, the AI Studio version of 1.5 Pro does have a 50 turn rate limit I believe).

[mcortt]

AI in sidebar is great. However, please give us some sort of keyboard shortcut to open it.

[BiORNADE]

You can put ChatGPT or any AI Chatting service into a sidebar, which makes the features useless and sort of a filler but... Actually, the only time that AI Chatbots in a browser was a big deal was when Microsoft Edge had BingAI, which was a free version of GPT 4.0 FE (with some improved enhancements), but it didn't last much until MS got Copilot (BingAI) in almost everywhere, so you can use it on Windows 7 now (sarcastically saying)

To test the waters with sidebar, using a known website (from time to time) is a nice idea, so the Firefox Team are able to detect the issue easier and deliver a fix earlier... But a native AI feature that works just like a sidebars is not that of an idea

But here's a nice thing; you can actually access everything just like watching a gecko video and commenting on this very forum using a simple technique.
Some more contrast in the lines, some more tests, some polish from the community, the final testing, then the final product. The technique could also extend to PWAs (the most Kudoed question still on review)

Looking ahead to an answer

Kind Regards,
BiORNADE

[Frisk]

Hello!

With introduction of even more AI services to Firefox I wanted to express that to me it does seem like Firefox is missing with development of features. This sentiment is echoed by a lot of people in my social bubble of technologists, ethicists and other people with same priorities as what one could think are values which Firefox was built on.

Those concerns are in my opinion very valid. The machine learning models have shown to be unreliable - just some of the recent examples from AI products made by large corporations: pointing users to eat pizza with glue, providing false information about just about anything. There are a number of ethical issues yet to be resolved with usage of AI, from its intense usage of computing resources that adds up to electric grid demands. Through privacy and possible copyright violations of datasets that power the models. To an entire bag of other issues monitored by excellent resources such as AIAAIC repository. 

AI/Machine learning is an amazing field with many likely applications, and yet, its recent rise to fame is characterized by failures and issues in many implementations. Personally I often don't even see whether the application of AI is truthfully necessary, in many cases a human would do a more trustworthy and fast task of information gathering than a large machine learning model.

When we compare current state of "AI", does it reflect what Firefox stands for? Does it reflect Mozilla's principals?

Let's compare.

• Principle 2

  The internet is a global public resource that must remain open and accessible.

LLMs are known for being a black box. Depending on our definition of "open and accessible", LLMs can be a very free resource or a completely inaccessible black box of math.

• Principle 4

  Individuals’ security and privacy on the internet are fundamental and must not be treated as optional.

It's clear that LLMs pose a privacy risk to Internet users. LLMs pose risk in at least two ways - because the data they are trained on sometimes contains private information due to negligent training process. In this case users of a learned model can possibly access private information. The second risk is of course usage of 3rd party services that may use information to infringe on privacy of users. While Mozilla in blog assures that "we are committed to following the principles of user choice, agency, and privacy as we bring AI-powered enhancements to Firefox", it's unclear how supporting such services as "ChatGPT, Google Gemini, HuggingChat, and Le Chat Mistral" helps protect Firefox user privacy. Giving users choice should not compromise their safety and privacy.

• Principle 10

  Magnifying the public benefit aspects of the internet is an important goal, worthy of time, attention and commitment.

In my opinion in the process of designing AI functionalities on top of Firefox there was no evaluation on how those functionalities can benefit the public. There are a number of issues as mentioned above with the LLMs, they can be dangerous and work in detriment to users. Investing and supporting in technology of this type may lead to terrible consequences with little actual benefit.

In addition Mozilla claims:

• We are committed to an internet that elevates critical thinking, reasoned argument, shared knowledge, and verifiable facts.

I argue that AI models are the opposite to that. AI output is not verifiable. They are working against sharing knowledge by making seemingly accurate information that turn out to be false.

I ask Mozilla to reevaluate impact of AI considering all of those points. I ask on behalf of myself as well as many users that I see on Fediverse being greatly worried and frustrated with AI changes added on top of Firefox. There is certainly a lot of potential greatness that could be done with AI, but those steps must be taken responsibly.

[narazamsa]

If its on device only and no data collection is going to happen. Also only complete opensource models. Then yes that would be good to have.

[ilikefirefox99]

I love the features being implemented, BUT... as many point out, a second sidebar  is necessary. The idea should be same as all other browsers. You have a vertical panel with your vertical tabs in the left, and you have the sidebar (history, bookmarks, etc, etc, etc) on the right.

Trying to create a single sidebar that also has your vertical tabs on it, it's a very messy concept and a terrible poor implementation in my view. I understand that it might be tricky to implement this feature, but honestly, it's very very very necessary.

No need to reinvent the wheel, Vivaldi, and especially Microsoft Edge, already show  you the best concept:

Thank you.

[0x4d6165]

if you're gonna use AI crap anyway why not just use edge lol

[Jon]

Thanks for the feedback! For those interested, there's another idea thread for a second sidebar here:

2nd Sidebar 

[ksetlak]

I have one observation: I'm logged in to Google on my work laptop and my work Google account is the primary one, while the private one is the secondary. However, I only use Gemini with my private account. I can't change the account I'm using in the side-docked window, because when I attempt to do that, a new tab is opened in my main Firefox window. Would be lovely if this was configurable and `authuser=1` could be added to the params in the URL used to open the side pane.

[0x4d6165]

Actually listen to your userbase for *once* when we say we do NOT want this! Be a good web browser stop trying to be edge.

[Reycko]

Would be nice to have a nice UI to add custom prompts if that's not in the works already

[aminought]

You can! Just edit "browser.ml.chat.prompt.prefix" and "browser.ml.chat.prompts.*" prefs. Also, you can add additional prompts like "Answer Questions" if you add "browser.ml.chat.prompts.3", "browser.ml.chat.prompts.4", etc.

[myspace]

Thanks going to try it out now!

[FastBrowser2022]

I tried using the right click > Simplify text option with Gemini. It worked the first time, but the 2nd time Gemini did a bot check. It otherwise worked fine after completing it.

I also tested it with Chat-GPT 4o, this worked for summarizing the same text and simplifying content.

However I noticed there is no context menu option to simply highlight text and re-write text in an unsubmitted text input field

[lackey]

Considering the privacy problems, corporate centralization, and environmental issues caused by the companies represented in Mozilla's AI chatbot integration, I have some potential recommendations, ranked from most to least palatable.

• Entirely discontinue this experiment and stop pursuing AI.
• Cannibalize the work done for integrating these 4 proprietary websites into Firefox, and use it for allowing users to add any website of their choice to the sidebar (like Floorp, Edge, or Vivaldi).
• Continue developing AI, but as a browser extension that is not shipped with Firefox.
• Continue developing AI, and ship it as a browser extension that can be fully erased by users if they so choose.

[advpod]

No.

Stop adding dumb trendy features. Spend engineering effort on integrating features that have nothing to do with browsing the web into a web browser. If I wanted to use a chatbot, I would install a chatbot app onto my computer. Features that have nothing to do with browsing the web have no place being in a web browser.

[eladio77]

Amazing feature. I find the funtion, but when i have active the vertical tabs i can not see the Boton "Chat IA", only i can see it without "Vertical Tabs" active. Can you add the shortcut access to the Vertical Tab menu?

[nclm]

Please Mozilla, don’t get lost. AI chatbots is a very silly feature, and it makes it hard to take you seriously. I know investors love it these days when tech companies integrate anything “AI” in their product, but you’re Mozilla, we all expect you to behave differently and for the good of the users and the web. There is no demand for AI chatbots, only unsolicited offer from the tech world, please don’t be part of this! There is nothing ethical about about offering a feature that is consistently giving misinformation based on stolen data processed on energy and water wasting machines. It would be lovely if you could focus your ressources on actually improving the browser and on communication campaigns to get more users, not on jumping on the latest fad. Thank you ❤️