Updated to v1.0.1

tanaikech · May 2, 2024 · 3ec4264 · 3ec4264
1 parent fc6e7c7
commit 3ec4264
Show file tree

Hide file tree

Showing 2 changed files with 273 additions and 5 deletions.
diff --git a/README.md b/README.md
@@ -37,6 +37,7 @@ I created this library based on the following reports.
 - [Generating Texts using Files Uploaded by Gemini 1.5 API](https://medium.com/google-cloud/generating-texts-using-files-uploaded-by-gemini-1-5-api-5777f1c902ab)
 - [Specifying Output Types for Gemini API with Google Apps Script](https://medium.com/google-cloud/specifying-output-types-for-gemini-api-with-google-apps-script-c2f6a753c8d7)
 - [Parsing Invoices using Gemini 1.5 API with Google Apps Script](https://medium.com/google-cloud/parsing-invoices-using-gemini-1-5-api-with-google-apps-script-1f32af1678f2)
+- [Taming the Wild Output: Effective Control of Gemini API Response Formats with response_mime_type](https://medium.com/google-cloud/taming-the-wild-output-effective-control-of-gemini-api-response-formats-with-response-mime-type-da273c08be85)
 
 # Features
 
@@ -64,6 +65,10 @@ Output Specification:
 
 - Specify the desired output format for the results generated by the Gemini API.
 
+- Using `response_mime_type` and JSON schema, the output format is controlled. [Ref](https://medium.com/google-cloud/taming-the-wild-output-effective-control-of-gemini-api-response-formats-with-response-mime-type-da273c08be85)
+
+# Features
+
 # Usage
 
 In order to test this script, please do the following steps.
@@ -162,8 +167,11 @@ The value of `object` is as follows.
 {Boolean} object.doCountToken Default is false. If this is true, when Gemini API is requested, the token of request is shown in the log.
 {Array} object.history History for continuing chat.
 {Array} object.functions If you want to give the custom functions, please use this.
+{String} object.response_mime_type In the current stage, only "application/json" can be used.
 ```
 
+When you want to use `response_mime_type`, please give `jsonSchema` to generateContent method. In the current stage, only `"application/json"` can be used to `response_mime_type`.
+
 <a name="setfileIds"></a>
 
 ## setFileIds
@@ -328,6 +336,49 @@ function myFunction() {
 }
 ```
 
+When you want to use `response_mime_type`, please give `jsonSchema` to generateContent method as follows. In this case, by giving only JSON schema, this library can return a valid object. You can also see the detailed information about `response_mime_type` at [my report](https://medium.com/google-cloud/taming-the-wild-output-effective-control-of-gemini-api-response-formats-with-response-mime-type-da273c08be85).
+
+```javascript
+function myFunction() {
+  const apiKey = "###"; // Please set your API key.
+
+  const g = GeminiWithFiles.geminiWithFiles({
+    apiKey,
+    response_mime_type: "application/json",
+  }); // This is for installing GeminiWithFiles as a library.
+  // const g = new GeminiWithFiles({ apiKey, response_mime_type: "application/json" }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
+
+  const jsonSchema = {
+    title: "5 popular cookie recipes",
+    description: "List 5 popular cookie recipes.",
+    type: "array",
+    items: {
+      type: "object",
+      properties: {
+        recipe_name: {
+          description: "Names of recipe.",
+          type: "string",
+        },
+      },
+    },
+  };
+  const res = g.generateContent({ jsonSchema });
+  console.log(res);
+}
+```
+
+When this script is run, the following result is obtained.
+
+```json
+[
+  { "recipe_name": "Chocolate Chip Cookies" },
+  { "recipe_name": "Peanut Butter Cookies" },
+  { "recipe_name": "Oatmeal Cookies" },
+  { "recipe_name": "Sugar Cookies" },
+  { "recipe_name": "Snickerdoodle Cookies" }
+]
+```
+
 ## Additional information
 
 ### Confirm current functions for the function calling
@@ -390,6 +441,7 @@ The sample scripts are as follows.
 - [Upload image files and create descriptions of images](#createdescriptons)
 - [Upload invoices of PDF data and parse them](#parseinvoices)
 - [Upload papers of PDF data and summarize them](#summarizepapers)
+- [Samples using response_mime_type](#samplesresponsemimetype)
 
 <a name="generatecontent"></a>
 
@@ -768,6 +820,198 @@ the following result was obtained by one API call. It is found that the uploaded
 ]
 ```
 
+<a name="samplesresponsemimetype"></a>
+
+## Samples using response_mime_type
+
+In the current stage, only `"application/json"` can be used to `response_mime_type`.
+
+### Sample 1
+
+```javascript
+function myFunction() {
+  const apiKey = "###"; // Please set your API key.
+  const g = GeminiWithFiles.geminiWithFiles({
+    apiKey,
+    doCountToken: true,
+    response_mime_type: "application/json",
+  }); // This is for installing GeminiWithFiles as a library.
+  // const g = new GeminiWithFiles({ apiKey, response_mime_type: "application/json" }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
+
+  const res1 = g.generateContent({ q: "What is Google Apps Script?" });
+  console.log(res1);
+}
+```
+
+In this case, the result is returned as an array as follows.
+
+```json
+[
+  "Google Apps Script is a cloud-based scripting platform that lets you integrate with and automate tasks across Google products like Gmail, Calendar, Drive, and more. It's based on JavaScript and provides easy ways to automate tasks across Google products and third-party services."
+]
+```
+
+### Sample 2
+
+```javascript
+function myFunction() {
+  const apiKey = "###"; // Please set your API key.
+
+  const g = GeminiWithFiles.geminiWithFiles({
+    apiKey,
+    doCountToken: true,
+    response_mime_type: "application/json",
+  }); // This is for installing GeminiWithFiles as a library.
+  // const g = new GeminiWithFiles({ apiKey, response_mime_type: "application/json" }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
+
+  // Question 1
+  const jsonSchema1 = {
+    title: "Current population of Kyoto, Osaka, Aichi, Fukuoka, Tokyo in Japan",
+    description:
+      "Return the current population of Kyoto, Osaka, Aichi, Fukuoka, Tokyo in Japan",
+    type: "object",
+    properties: {
+      propertyNames: {
+        description: "Prefecture names",
+      },
+      patternProperties: {
+        "": { type: "number", description: "Population" },
+      },
+    },
+  };
+  const res1 = g.generateContent({ jsonSchema: jsonSchema1 });
+  console.log(res1);
+
+  // Question 2
+  const jsonSchema2 = {
+    title: "Current area of them",
+    description: "Return the current area of them.",
+    type: "object",
+    properties: {
+      propertyNames: {
+        description: "Prefecture names",
+      },
+      patternProperties: {
+        "": { type: "number", description: "Area. Unit is km^2." },
+      },
+    },
+  };
+  const res2 = g.generateContent({ jsonSchema: jsonSchema2 });
+  console.log(res2);
+}
+```
+
+In this case, the result values can be obtained by giving only JSON schema. The result is as follows.
+
+For 1st question
+
+```json
+{
+  "Kyoto": 2579970,
+  "Osaka": 8837684,
+  "Aichi": 7552873,
+  "Fukuoka": 5138217,
+  "Tokyo": 14047594
+}
+```
+
+For 2nd question
+
+```json
+{
+  "Kyoto": 4612.19,
+  "Osaka": 1904.99,
+  "Aichi": 5172.92,
+  "Fukuoka": 4986.51,
+  "Tokyo": 2194.07
+}
+```
+
+### Sample 2
+
+```javascript
+async function myFunction() {
+  const apiKey = "###"; // Please set your API key.
+
+  // Please set file IDs of PDF file of invoices.
+  const fileIds = ["###fileID1###", "###fileID2###"];
+
+  const jsonSchema = {
+    title:
+      "Array including JSON object parsed the following images of the invoices",
+    description:
+      "Create an array including JSON object parsed the following images of the invoices.",
+    type: "array",
+    items: {
+      type: "object",
+      properties: {
+        name: {
+          description: "Name given as 'Filename'",
+          type: "string",
+        },
+        invoiceTitle: {
+          description: "Title of invoice",
+          type: "string",
+        },
+        invoiceDate: {
+          description: "Date of invoice",
+          type: "string",
+        },
+        invoiceNumber: {
+          description: "Number of the invoice",
+          type: "string",
+        },
+        invoiceDestinationName: {
+          description: "Name of destination of invoice",
+          type: "string",
+        },
+        invoiceDestinationAddress: {
+          description: "Address of the destination of invoice",
+          type: "string",
+        },
+        totalCost: {
+          description: "Total cost of all costs",
+          type: "string",
+        },
+        table: {
+          description:
+            "Table of invoice. This is a 2-dimensional array. Add the first header row to the table in the 2-dimensional array. The column should be 'title or description of item', 'number of items', 'unit cost', 'total cost'",
+          type: "array",
+        },
+      },
+      required: [
+        "name",
+        "invoiceTitle",
+        "invoiceDate",
+        "invoiceNumber",
+        "invoiceDestinationName",
+        "invoiceDestinationAddress",
+        "totalCost",
+        "table",
+      ],
+      additionalProperties: false,
+    },
+  };
+
+  const g = GeminiWithFiles.geminiWithFiles({
+    apiKey,
+    doCountToken: true,
+    response_mime_type: "application/json",
+  }); // This is for installing GeminiWithFiles as a library.
+  // const g = new GeminiWithFiles({ apiKey, doCountToken: true, response_mime_type: "application/json" }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
+  const fileList = await g.setFileIds(fileIds, true).uploadFiles();
+  const res = g
+    .withUploadedFilesByGenerateContent(fileList)
+    .generateContent({ jsonSchema });
+  // g.deleteFiles(fileList.map(({ name }) => name)); // If you want to delete the uploaded files, please use this.
+  console.log(JSON.stringify(res));
+}
+```
+
+When this script is run to the same invoices of [the section "Upload invoices of PDF data and parse them"](#parseinvoices), the same result is obtained.
+
+If you want to return the value of High-complexity JSON schemas, `response_mime_type` might be suitable.
+
 # IMPORTANT
 
 - If an error occurs, please try again after several minutes.
@@ -812,4 +1056,8 @@ I have already proposed the following future requests to the Google issue tracke
 
   1. Initial release.
 
+- v1.0.1 (May 2, 2024)
+
+  1. `response_mime_type` got to be able to be used for controlling the output format. [Ref](#samplesresponsemimetype)
+
 [TOP](#top)
diff --git a/classGeminiWithFiles.js b/classGeminiWithFiles.js
@@ -17,9 +17,10 @@ class GeminiWithFiles {
    * @param {Boolean} object.doCountToken Default is false. If this is true, when Gemini API is requested, the token of request is shown in the log.
    * @param {Array} object.history History for continuing chat.
    * @param {Array} object.functions If you want to give the custom functions, please use this.
+   * @param {String} object.response_mime_type In the current stage, only "application/json" can be used.
    */
   constructor(object = {}) {
-    const { apiKey, accessToken, model, version, doCountToken, history, functions } = object;
+    const { apiKey, accessToken, model, version, doCountToken, history, functions, response_mime_type } = object;
 
     /** @private */
     this.model = model || "models/gemini-1.5-pro-latest";
@@ -74,6 +75,9 @@ class GeminiWithFiles {
     /** @private */
     this.fileList = [];
 
+    /** @private */
+    this.response_mime_type = "";
+
     /**
      * Functions for function calling of Gemini API. You can see the default functions as follows. You can create the value of functions by confirming the default values.
      * 
@@ -167,7 +171,10 @@ class GeminiWithFiles {
       customType_array: (e) => e.items,
       customType_object: (e) => e.items,
     };
-
+    if (response_mime_type && response_mime_type != "") {
+      this.response_mime_type = response_mime_type;
+      this.functions = {};
+    }
     if (functions && functions.params_) {
       this.functions = functions;
     }
@@ -402,10 +409,13 @@ class GeminiWithFiles {
     if (!object || typeof object != "object") {
       throw new Error("Please set object including question.");
     }
-    let { q } = object;
-    if (!q || q === "") {
+    let { q, jsonSchema } = object;
+    if ((!q || q === "") && (!jsonSchema || typeof jsonSchema != "object")) {
       throw new Error("Please set a question.");
     }
+    if (!q || q === "" && (jsonSchema || typeof jsonSchema == "object")) {
+      q = `Follow JSON schema.<JSONSchema>${JSON.stringify(jsonSchema)}</JSONSchema>`;
+    }
     let uploadedFiles = this.fileList.length > 0 ? this.fileList : [];
     if (uploadedFiles.length > 0) {
       const n = uploadedFiles.reduce((n, o) => (n += o.files ? o.files.length : 1), 0);
@@ -455,6 +465,9 @@ class GeminiWithFiles {
     do {
       retry--;
       const payload = { contents, tools: [{ function_declarations }] };
+      if (this.response_mime_type != "") {
+        payload.generationConfig = { response_mime_type: this.response_mime_type };
+      }
       if (this.doCountToken) {
         const res = this.fetch_({
           url: this.addQueryParameters_(this.urlCountToken, this.queryParameters),
@@ -531,7 +544,14 @@ class GeminiWithFiles {
       console.warn(output);
       return "No values.";
     }
-    return output.text.trim();
+    // return output.text.trim();
+    const returnValue = output.text.trim();
+    try {
+      return JSON.parse(returnValue);
+    } catch (stack) {
+      console.warn(stack);
+      return returnValue;
+    }
   }
 
   /**