Google Gemini — A New Era in Generative AI

13 min readDec 29, 2023

In the ever-evolving landscape of artificial intelligence (AI), Google has taken a significant leap forward with the introduction of Google Gemini. This cutting-edge technology, developed by DeepMind, promises to redefine the capabilities of Generative AI models, offering a host of features and integrations that aim to push the boundaries of what is possible.

Let’s delve into the various aspects of Google Gemini, exploring its features, capabilities, integrations, models, and future plans. Additionally, we will provide a comparative analysis with OpenAI’s ChatGPT 4 to highlight the unique strengths and characteristics of each.

Capabilities

Google Gemini demonstrates exceptional capabilities in generating high-quality and contextually relevant responses. Its training on diverse datasets and advanced algorithms contribute to its ability to understand user queries, even in complex or ambiguous contexts. The model’s adaptability allows it to excel in various domains, from general knowledge queries to specialized industry-specific conversations.

Key highlights:

Massive Multitask Language Understanding: Gemini achieves a groundbreaking milestone by surpassing human experts in MMLU, a prominent evaluation method that gauges the knowledge and problem-solving capabilities of AI models. While Human Experts scored 89.8% accuracy, the state-of-the-art (SOTA) GPT-4 fell short by a huge margin at 86.4% accuracy only. On the other hand, Gemini surpassed even Human Experts by achieving a score of 90% in MMLU tasks.
Multiple benchmarks: Gemini also outperformed and surpassessed state-of-the-art performance of GPT-4 on a range of benchmarks including text (MMLU), reasoning (Big-bench Hard, HellaSwag, etc.), math (GSM8K and MATH), coding (HumanEval and Natural2Code).
Truly Natively Multimodal: Furthermore, it outperformed the SOTA competitor, GPT-4, in all multimodal domains including image, video, and audio. According to Google, Gemini Ultra surpassed in many benchmarks including MMMU, VQAv2, TextVQA, Infographic VQA, among others for image understanding, VATEX and perception Test MCQA for Video, CoVoST2 and FLEURS for multiple languages.

Features

Google Gemini boasts of a rich feature set, distinguishing it in the competitive AI landscape.

One notable feature is its advanced natural language understanding, enabling more nuanced and contextually aware interactions. This enhancement is a result of extensive training on diverse datasets, allowing Gemini to grasp the subtleties of human communication.

To give you a glimpse of how Gemini is revolutionizing the Gen AI landscape, sample this. As of December, Gemini Pro is now available with Bard and is connected with other Google apps like Flights and Hotels. This gives Bard the ability to help you get real-time information.

We entered the following prompt in Bard:

I’m traveling from NY to LA next week for the New Year’s celebrations. Give me details of some flight options and a decent mid-premium segment hotel option for the duration of Dec 30 to Jan 2.

We were amazed when Bard gave us the following details:

The Gemini models also introduce novel capabilities in multitasking, allowing users to seamlessly switch between tasks and contexts within a conversation. This feature enhances the user experience by making interactions with the AI more dynamic and versatile.

Some more notable features are:

Gemini supports multimodal dialog through text prompts and even audio and video inputs.
It supports major multiple natural languages, for translation and summarization tasks, with the support for more to come.
Gemini can generate code based on different inputs you give it — be it text prompt or a visual.
It can generate text and image combined — like a pro copywriter or a storyteller!
Gemini can translate visuals into speech and text for analysis, instruction, learning, and sharing.
It can reason visually in multiple languages and solve visual puzzles for you converging entertainment with learning.
Create new games with clear rules for playing for fun and boosting your brainpower.
It can find logical and natural connections in even unseemingly disconnected entities.
It supports complex logical and spatial reasoning allowing more use cases for location-based services.
Gemini is built with responsibility ingrained from the beginning. Proper safeguards and guardrails are incorporated in the model, along with partners making it more inclusive, fair, and safe.
State-of-the-art encryption and anonymization mechanisms ensure privacy, user data protection, and confidentiality of all user interactions.

Gemini comes in three sizes

Gemini Nano

The most portable and the most efficient model optimized for mobile devices for running on-device tasks. Nano runs directly on mobile processors allowing developers to reimagine a new range of extraordinary use cases.

The biggest advantage of Nano running directly on the device is that it can even process data that should not be shared or transmitted including suggestions for message replies in a messenger. On-device feature allows Gemini Nano deliver a consistent experience within a range-bound latency — even in absence of network!

Gemini Nano packs powerful features such as advanced proofreading, grammar checking, text summarization, and context-based smart replies. Though the developer SDKs and APIS are still awaited, Google’s Pixel 8 Pro now runs Gemini Nano allowing its users to summarize recorded calls in close to 30 languages!

Gemini Pro

This is Google’s best offering for allowing scaling of a wide range of AI tasks. As of December 6, 2023, a specially tuned model of Gemini Pro is already integrated in Google’s Gen AI chatbot Bard.

Gemini Pro was made available for developers and enterprises through the Gemini API that is accessible on Google’s AI Studio and Vertex AI on cloud.

The Pro model easily surpasses all currently available similarly-sized models on major research benchmarks.
It supports a 32K context window for text with a larger window in future versions.
As of now, Gemini Pro is available for use, within limits, free of cost and Google promises to competitively price it going forward.
The Gemini Pro API supports features that every developer loves to have in a Gen AI model — chat functionality, function calling, storing embeddings, semantic retrieval, and custom grounding of enterprise knowledge.
As of now, 38 languages are supported in more than 180 geographies.
The API supports the conventional text-to-text semantic with a Vision multimodal endpoint allowing acceptance of input as text and images and generating text output.
Gemini Pro SDKs are available in Python, Android (Kotlin), Node.js, Swift and JavaScript allowing you to build apps that can run anywhere.

Gemini Ultra

Google claims that Gemini Ultra is the most advanced, capable, and the largest model in the Gemini family suited for complex tasks. As per the report released by Google DeepMind, its capabilities include advanced SOTA in large-scale language modeling, image understanding, audio processing, and video understanding in a wide range of language, coding, reasoning, and multimodal tasks. Though, we must confess that details about its prowess and capabilities are still fuzzy.

Before making it broadly available, Google is in the final stage of running extensive safety checks, implementing guardrails for Gemini Ultra, for state-of-the-art privacy and trust. It is done with the help of trusted partners for tasks such as red-teaming, fine-tuning, and reinforcement learning from human feedback (RLHF).

Initially Gemini Ultra will be available only to select clients, developers, and partners. This will include experts in the field of AI safety and responsibility for early experimentation, feedback, and further improvements.

With the launch of Bard Advanced early next year, Gemini Ultra will be widely available for all developers and enterprise customers giving you access to the best models and capabilities.

So, we still have something to lookout for!

Integrations

The integration capabilities of Google Gemini extend its reach across multiple platforms and applications. Coming from Google’s stable there is no doubt it will have seamless integrations with Google Workspace, Cloud, and even Android. The availability of Gemini API and SDKs in multiple languages/platforms allow it to be leveraged in many more platforms and environments.

Some of the major integrations that you can look forward to are:

Android developers, in particular, can leverage the power of generative AI in their applications with the help of Google AI Studio and Google AI SDK in Kotlin. The new version of Android Studio will come equipped with integration of Gemini integration capabilities. Now you can allow your users to experience the power of AI and open up new possibilities for creative and dynamic user experiences in Android apps.
The Pixel Feature Drop in December 2023 further emphasizes the integration potential of Gemini Nano for on-device tasks. The update introduces features that harness the AI’s capabilities to enhance user interactions on Pixel devices. Some of the innovative features include Recorder summarization, smart replies for Gboard, video boost for Camera for enhanced shooting experience, Dual Screen preview on Pixel Fold, cleaning scanned documents, Repair Mode for peace of mind, and many more.
Developers looking to try out Gemini Pro can use the API via AI Studio, a free web-based developer tool. When they’re ready for a fully-managed AI platform, developers can easily transition their AI Studio code to Vertex AI for additional customization and Google Cloud features. Using Google’s world class unified AI Stack helps you reap the benefits of planetary scale AI infrastructure, top-notch models, and access to Vertex AI and Duet AI to develop AI-powered solutions for enterprises at scale.

Comparison with OpenAI’s ChatGPT

When Google launched Gemini, in the first few hours it was being dubbed as the “ChatGPT-4 Killer!” Now we don’t know if Gemini is indeed the OpenAI’s ChatGPT (or its model GPT-4, backed by Microsoft) killer or not, but it begs a comparison between them.

Both Google Gemini and OpenAI’s GPT excel in natural language understanding and generation, but they have distinct characteristics.

Google’s Own Claims

Gemini’s emphasis on real-time learning and multitasking sets it apart from ChatGPT 4. While both models exhibit impressive capabilities, Gemini’s dynamic adaptation to user interactions gives it an edge in certain scenarios. On the other hand, ChatGPT 4 is recognized for its extensive pre-training on diverse datasets, leading to a broad understanding of various topics.

The performance of Gemini Ultra, the largest of the models in the Gemini family, and its comparison with the previous leader, the GPT-4, are as per the figures reported by Google. As of now, we cannot verify the same from independent sources, so keep this caveat in mind going forward.

Table: A summary of differences between Google Gemini and OpenAI GPT class of models.

Content Analysis and Generation

In addition to the above points of differences, comparing the two models on one of the most common tasks of content generation and analysis, is something that will require a lot of experimentation.

Aaron Mok, ran a few tests on both Gemini and GPT for the most common use cases that we are already thinking of and deploying in our daily lives. These included identifying if some content is AI or Human generated, describing an image, asking some sexually graphic questions (to see if the model blocks you entirely or generates an educated response to guide the user), asking information based on latest developments, writing a resignation letter, latest geopolitical conflicts, article summarization, and so on.

As you can imagine, no one model is superior to the other in all, or even most counts. One model, Gemini, was better at detecting an AI-generated image, but the other was better at describing it. Similarly, while GPT-4 generated a more human-like resignation letter, Gemini’s output was more concise and to the point.

An Independent and Objective Comparison: Gemini Pro vs GPT-3.5 Turbo

Researchers at Carnegie Mellon University and BerriAI have conducted benchmark tests on two comparable models from Google and OpenAI — Gemini Pro and GPT-3.5 turbo. The aim was to conduct a third-party, independent, and objective comparison of the two models with reproducible code and transparent results for peer review. With this research, they took an in-depth view of the abilities of the two models in the light of the results produced and pointed out the areas where one of them surpasses the other.

Researchers also evaluated GPT-4 Turbo to the mix to compare its results with both the above models. They also added the recently released Mixtral model, a new Sparse Mixture-of-Experts class of generative AI model from Mistral AI team, to a subset of the tests. For the purpose of this article, we will ignore the evaluation results for Mixtral.

To achieve consistency in the experiments, LiteLLM4’ unified interface was used to query the models between December 11–15, 2023. Google vertex Ai was used for querying Gemini Pro, while OpenAI API was used for querying the two GPT models. The pricing of the models was also considered for the purpose of ascertaining the financial impact of deploying the models. The prices for Gemini Pro and GPT-3.5 Turbo were comparable during the experiment duration.

Table: Gemini Pro charges by character; so a rule-of-thumb of 4 characters per English token is used to arrive at its cost.

The research performed tests on 10 publicly available datasets to evaluate a wide range of language understanding, processing, and generating abilities. These tests included areas such as question-answering, natural language translation, reasoning, generating code, solving math problems, and the models’ ability to act as an instruction-following agent.

For a fair comparison, the researchers ran consistent experiments on all models — they used exactly the same prompts and applied the same evaluation protocols for all models under test. It ensured that all models get exactly the same input as a level-playing field unlike previous research where the experiment settings may differ.

The prompts as well as the evaluators were both taken from standard repositories — mostly from the officially released datasets or from publicly available Eleuther evaluation harness. All prompts typically consist of a mandatory query and input, and optional few-shot examples or a chain-of-thought reasoning. Even when minor deviations from standard prompts/evaluators were made, they were kept consistent across all test subjects and are well-documented.

Here are the overall results of the benchmark evaluation comparing Gemini Pro, GPT-3.5 Turbo, and GPT-4 Turbo.

Table Notes:

The actual evaluation results (for Gemini Pro, GPT-3.5 Turbo, and GPT-4 Turbo) are taken as-is from the source, with the evaluation data for the Mixtral model omitted.
The difference in the model performance is calculated by us for a better understanding of the evaluation results.
The better model among Gemini Pro and GPT-3.5 Turbo is shown in green and the other one in red.
A darker shade of green represents that the model was the best performer among all three — Gemini Pro, GPT-3.5 Turbo, and GPT-4 Turbo.
The dark blue cell in the difference represents the highest gap in performance and the light blue indicates the least gap in performance between Gemini Pro and GPT-3.5 Turbo.

Here we present the results in the form of a chart for easier understanding of the readers.

Figure: Evaluation results for Gemini Pro and GPT-3.5 Turbo and the difference in their accuracy.

It is evident from the results that Gemini Pro is not able to surpass GPT-3.5 Turbo in any of the benchmark evaluations. Although it is quite close in accuracy to the incumbent leader, on most counts, the gap is significant in some tasks such as CoT-based Question answering, acting as an agent, code generation, and language translation.

The research also offers an explanation about the under-performance of Gemini Pro on multiple counts — for e.g., for mathematical reasoning with many digits, heightened sensitivity to the order of multiple-choice answers, and Gemini’s aggressive content filtering protocols. The research also points out categorically Gemini’s superiority over GPT-3.5 Turbo in generating non-English language text content and its ability to better handle more complex and longer reasoning chains in prompts.

After this report was published on December 19, 2023, the hype around Gemini Pro has subsided quite a bit, and understandably so. AI enthusiasts and researchers are now eagerly waiting for Google to make available not only the details of their comparison analysis as well as the release of Gemini Ultra in its full capacity to make a more informed choice.

In conclusion, the jury on the debate between the superiority of Google’s Gemini and OpenAI’s GPT models is still out. As of now, people are only discovering the specific use cases with Gemini Pro, while GPT’s models are quite familiar to the community. We would just add that both models contribute significantly to the advancement of AI, offering unique features that cater to diverse user needs.

Conclusion

In summary, Google Gemini emerges as a powerful addition to the AI landscape, with its advanced features, capabilities, and integrations. As it continues to evolve through user feedback and updates, Gemini holds promise for reshaping the way we interact with AI.

The comparative analysis with OpenAI’s ChatGPT 4 underscores the diversity and richness of the AI ecosystem, with each model bringing its own strengths to the table. The future of AI appears vibrant and dynamic, with Google Gemini leading the charge in innovation and user-centric AI development.