{"_id":"5b646196166953000321caa8","project":"5633ebff7e9e880d00af1a53","version":{"_id":"5a8fae0268264c001f20cc00","project":"5633ebff7e9e880d00af1a53","__v":4,"createdAt":"2018-02-23T06:00:34.961Z","releaseDate":"2018-02-23T06:00:34.961Z","categories":["5a8fae0268264c001f20cc01","5a8fae0268264c001f20cc02","5a8fae0368264c001f20cc03","5a8fae0368264c001f20cc04","5a8fae0368264c001f20cc05","5a8fae0368264c001f20cc06","5a8fae0368264c001f20cc07","5a8fae0368264c001f20cc08","5a8fae0368264c001f20cc09","5abaa7eb72d6dc0028a07bf3","5b8ee7842790f8000333f9ba","5b8ee8f244a21a00034b5cd9"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"2.0.0","version":"2.0"},"category":{"_id":"5a8fae0268264c001f20cc02","version":"5a8fae0268264c001f20cc00","project":"5633ebff7e9e880d00af1a53","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-12-23T20:06:16.013Z","from_sync":false,"order":4,"slug":"quickstart-guides-1","title":"Quickstart Guides"},"user":"5ab13dd42459e0004e761001","githubsync":"","__v":0,"parentDoc":null,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2018-08-03T14:07:18.670Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":11,"body":"[Hugo](https://gohugo.io/) is a static site generator written in Go. It is conceptually similar to [Jekyll](https://jekyllrb.com/), albeit with far more speed and flexibility. Hugo also supports generating [output formats](https://gohugo.io/templates/output-formats/) other than HTML, which allows users to pipe content directly into an Elasticsearch cluster.\n\nIn this guide, we are going to use this feature to tell Hugo to generate the exact format needed to submit the file to the _bulk endpoint of Elasticsearch.\n[block:api-header]\n{\n  \"title\": \"First Steps\"\n}\n[/block]\nIn order to make use of this documentation, you will need Hugo installed and configured on your system\n\n1. **Make Sure You Have Hugo Installed**. This guide assumes you already have Hugo installed and and configured on your system. Visit the [Hugo Documentation](https://gohugo.io/getting-started/) to get started.\n2. **Spin Up a Bonsai Elasticsearch Cluster**. This guide will use a Bonsai cluster as the Elasticsearch backend. Guides are available for setting up a cluster through [bonsai.io](doc:getting-started-with-bonsaiio), [Heroku](doc:getting-started-with-heroku), and [Manifold](doc:getting-started-with-manifold).\n3. **Create an Index on the Cluster**. In this example, we're going to push data into an index called `hugo`. This index needs to be created before any data can be stored on it. The index can be created either through the [Interactive Console](doc:managing-your-cluster#interactive-console), or with a tool like `curl`:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"# Use the URL for your cluster. A Bonsai URL looks something like this:\\ncurl -XPUT https://user123:pass456:::at:::my-awesome-cluster-1234.us-east-1.bonsai.io/hugo\",\n      \"language\": \"curl\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"title\": \"Configure Hugo to Output to Bonsai Elasticsearch\"\n}\n[/block]\nHugo's configuration settings live in a file called `config.toml` by default. This file may also have a `.json` or `.yaml`/`yml` extension. Add the following snippet based on your config file format:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"[outputs]\\nhome = [\\\"HTML\\\", \\\"RSS\\\", \\\"Bonsai\\\"]\\n\\n[outputFormats.Bonsai]\\nbaseName = \\\"bonsai\\\"\\nisPlainText = true\\nmediaType = \\\"application/json\\\"\\nnotAlternative = true\\n\\n[params.bonsai]\\nvars = [\\\"title\\\", \\\"summary\\\", \\\"date\\\", \\\"publishdate\\\", \\\"expirydate\\\", \\\"permalink\\\"]\\nparams = [\\\"categories\\\", \\\"tags\\\"]\",\n      \"language\": \"toml\"\n    },\n    {\n      \"code\": \"{\\n  \\\"outputs\\\": {\\n    \\\"home\\\": [\\n      \\\"HTML\\\",\\n      \\\"RSS\\\",\\n      \\\"Bonsai\\\"\\n    ]\\n  },\\n  \\\"outputFormats\\\": {\\n    \\\"Bonsai\\\": {\\n      \\\"baseName\\\": \\\"bonsai\\\",\\n      \\\"isPlainText\\\": true,\\n      \\\"mediaType\\\": \\\"application/json\\\",\\n      \\\"notAlternative\\\": true\\n    }\\n  },\\n  \\\"params\\\": {\\n    \\\"bonsai\\\": {\\n      \\\"vars\\\": [\\n        \\\"title\\\",\\n        \\\"summary\\\",\\n        \\\"date\\\",\\n        \\\"publishdate\\\",\\n        \\\"expirydate\\\",\\n        \\\"permalink\\\"\\n      ],\\n      \\\"params\\\": [\\n        \\\"categories\\\",\\n        \\\"tags\\\"\\n      ]\\n    }\\n  }\\n}\",\n      \"language\": \"json\"\n    },\n    {\n      \"code\": \"outputs:\\n  home:\\n    - HTML\\n    - RSS\\n    - Bonsai\\noutputFormats:\\n  Bonsai:\\n    baseName: bonsai\\n    isPlainText: true\\n    mediaType: application/json\\n    notAlternative: true\\nparams:\\n  bonsai:\\n    vars:\\n      - title\\n      - summary\\n      - date\\n      - publishdate\\n      - expirydate\\n      - permalink\\n    params:\\n      - categories\\n      - tags\\n\",\n      \"language\": \"yaml\",\n      \"name\": null\n    }\n  ]\n}\n[/block]\nThis snippet defines a new output called \"Bonsai\", and specifies some associated variables.\n[block:api-header]\n{\n  \"title\": \"Creating the JSON template\"\n}\n[/block]\nHugo needs to have a template for rendering data in a way that Elasticsearch will understand. To do this, we will define a JSON template that conforms to the [Elasticsearch Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html).\n\nCreate a template called `layouts/_default/list.bonsai.json` and give it the following content:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{{/* Generates a valid Elasticsearch _bulk index payload */}}\\n{{- $section := $.Site.GetPage \\\"section\\\" .Section }}\\n{{- range .Site.AllPages -}}\\n  {{- if or (and (.IsDescendant $section) (and (not .Draft) (not .Params.private))) $section.IsHome -}}\\n    {{/* action / metadata */}}\\n    {{ (dict \\\"index\\\" (dict \\\"_index\\\" \\\"hugo\\\" \\\"_type\\\" \\\"doc\\\"  \\\"_id\\\" .UniqueID)) | jsonify }}\\n    {{ (dict \\\"objectID\\\" .UniqueID \\\"date\\\" .Date.UTC.Unix \\\"description\\\" .Description \\\"dir\\\" .Dir \\\"expirydate\\\" .ExpiryDate.UTC.Unix \\\"fuzzywordcount\\\" .FuzzyWordCount \\\"keywords\\\" .Keywords \\\"kind\\\" .Kind \\\"lang\\\" .Lang \\\"lastmod\\\" .Lastmod.UTC.Unix \\\"permalink\\\" .Permalink \\\"publishdate\\\" .PublishDate \\\"readingtime\\\" .ReadingTime \\\"relpermalink\\\" .RelPermalink \\\"summary\\\" .Summary \\\"title\\\" .Title \\\"type\\\" .Type \\\"url\\\" .URL \\\"weight\\\" .Weight \\\"wordcount\\\" .WordCount \\\"section\\\" .Section \\\"tags\\\" .Params.Tags \\\"categories\\\" .Params.Categories \\\"authors\\\" .Params.Authors) | jsonify }}\\n  {{- end -}}\\n{{- end }}\",\n      \"language\": \"go\"\n    }\n  ]\n}\n[/block]\nWhen the site is generated, this will result in creating a file called `public/bonsai.json`, which will have the content stored in a way that can be pushed directly into Elasticsearch using the Bulk API.\n[block:api-header]\n{\n  \"title\": \"Push the Data Into Elasticsearch\"\n}\n[/block]\nTo get the site's data into Elasticsearch, render it by running `hugo` on the command line. Then send it to your Bonsai cluster with `curl`:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"curl -H \\\"Content-Type: application/x-ndjson\\\" -XPOST \\\"https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_bulk\\\" --data-binary @public/bonsai.json\",\n      \"language\": \"curl\"\n    }\n  ]\n}\n[/block]\nYou should now be able to see your data in the Elasticsearch cluster:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"$ curl -XGET \\\"https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_search\\\"\\n\\n{\\\"took\\\":1,\\\"timed_out\\\":false,\\\"_shards\\\":{\\\"total\\\":2,\\\"successful\\\":2,\\\"failed\\\":0},\\\"hits\\\":{\\\"total\\\":1,\\\"max_score\\\":1.0,\\\"hits\\\":[{\\\"_index\\\":\\\"hugo\\\",\\\"_type\\\":\\\"doc\\\",\\\"_id\\\":...\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]","excerpt":"Configure Hugo to Ingest into Elasticsearch","slug":"hugo","type":"basic","title":"Hugo"}

Bonsai.io Basics

Bonsai Heroku Basics

Bonsai Heroku Advanced

Bonsai Manifold Basics

Quickstart Guides

Bonsai.io Platform

Elasticsearch Tips & Tricks

Common Tools

Troubleshooting

Compliance

Hugo

Configure Hugo to Ingest into Elasticsearch
[Hugo](https://gohugo.io/) is a static site generator written in Go. It is conceptually similar to [Jekyll](https://jekyllrb.com/), albeit with far more speed and flexibility. Hugo also supports generating [output formats](https://gohugo.io/templates/output-formats/) other than HTML, which allows users to pipe content directly into an Elasticsearch cluster. In this guide, we are going to use this feature to tell Hugo to generate the exact format needed to submit the file to the _bulk endpoint of Elasticsearch. [block:api-header] { "title": "First Steps" } [/block] In order to make use of this documentation, you will need Hugo installed and configured on your system 1. **Make Sure You Have Hugo Installed**. This guide assumes you already have Hugo installed and and configured on your system. Visit the [Hugo Documentation](https://gohugo.io/getting-started/) to get started. 2. **Spin Up a Bonsai Elasticsearch Cluster**. This guide will use a Bonsai cluster as the Elasticsearch backend. Guides are available for setting up a cluster through [bonsai.io](doc:getting-started-with-bonsaiio), [Heroku](doc:getting-started-with-heroku), and [Manifold](doc:getting-started-with-manifold). 3. **Create an Index on the Cluster**. In this example, we're going to push data into an index called `hugo`. This index needs to be created before any data can be stored on it. The index can be created either through the [Interactive Console](doc:managing-your-cluster#interactive-console), or with a tool like `curl`: [block:code] { "codes": [ { "code": "# Use the URL for your cluster. A Bonsai URL looks something like this:\ncurl -XPUT https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/hugo", "language": "curl" } ] } [/block] [block:api-header] { "title": "Configure Hugo to Output to Bonsai Elasticsearch" } [/block] Hugo's configuration settings live in a file called `config.toml` by default. This file may also have a `.json` or `.yaml`/`yml` extension. Add the following snippet based on your config file format: [block:code] { "codes": [ { "code": "[outputs]\nhome = [\"HTML\", \"RSS\", \"Bonsai\"]\n\n[outputFormats.Bonsai]\nbaseName = \"bonsai\"\nisPlainText = true\nmediaType = \"application/json\"\nnotAlternative = true\n\n[params.bonsai]\nvars = [\"title\", \"summary\", \"date\", \"publishdate\", \"expirydate\", \"permalink\"]\nparams = [\"categories\", \"tags\"]", "language": "toml" }, { "code": "{\n \"outputs\": {\n \"home\": [\n \"HTML\",\n \"RSS\",\n \"Bonsai\"\n ]\n },\n \"outputFormats\": {\n \"Bonsai\": {\n \"baseName\": \"bonsai\",\n \"isPlainText\": true,\n \"mediaType\": \"application/json\",\n \"notAlternative\": true\n }\n },\n \"params\": {\n \"bonsai\": {\n \"vars\": [\n \"title\",\n \"summary\",\n \"date\",\n \"publishdate\",\n \"expirydate\",\n \"permalink\"\n ],\n \"params\": [\n \"categories\",\n \"tags\"\n ]\n }\n }\n}", "language": "json" }, { "code": "outputs:\n home:\n - HTML\n - RSS\n - Bonsai\noutputFormats:\n Bonsai:\n baseName: bonsai\n isPlainText: true\n mediaType: application/json\n notAlternative: true\nparams:\n bonsai:\n vars:\n - title\n - summary\n - date\n - publishdate\n - expirydate\n - permalink\n params:\n - categories\n - tags\n", "language": "yaml", "name": null } ] } [/block] This snippet defines a new output called "Bonsai", and specifies some associated variables. [block:api-header] { "title": "Creating the JSON template" } [/block] Hugo needs to have a template for rendering data in a way that Elasticsearch will understand. To do this, we will define a JSON template that conforms to the [Elasticsearch Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html). Create a template called `layouts/_default/list.bonsai.json` and give it the following content: [block:code] { "codes": [ { "code": "{{/* Generates a valid Elasticsearch _bulk index payload */}}\n{{- $section := $.Site.GetPage \"section\" .Section }}\n{{- range .Site.AllPages -}}\n {{- if or (and (.IsDescendant $section) (and (not .Draft) (not .Params.private))) $section.IsHome -}}\n {{/* action / metadata */}}\n {{ (dict \"index\" (dict \"_index\" \"hugo\" \"_type\" \"doc\" \"_id\" .UniqueID)) | jsonify }}\n {{ (dict \"objectID\" .UniqueID \"date\" .Date.UTC.Unix \"description\" .Description \"dir\" .Dir \"expirydate\" .ExpiryDate.UTC.Unix \"fuzzywordcount\" .FuzzyWordCount \"keywords\" .Keywords \"kind\" .Kind \"lang\" .Lang \"lastmod\" .Lastmod.UTC.Unix \"permalink\" .Permalink \"publishdate\" .PublishDate \"readingtime\" .ReadingTime \"relpermalink\" .RelPermalink \"summary\" .Summary \"title\" .Title \"type\" .Type \"url\" .URL \"weight\" .Weight \"wordcount\" .WordCount \"section\" .Section \"tags\" .Params.Tags \"categories\" .Params.Categories \"authors\" .Params.Authors) | jsonify }}\n {{- end -}}\n{{- end }}", "language": "go" } ] } [/block] When the site is generated, this will result in creating a file called `public/bonsai.json`, which will have the content stored in a way that can be pushed directly into Elasticsearch using the Bulk API. [block:api-header] { "title": "Push the Data Into Elasticsearch" } [/block] To get the site's data into Elasticsearch, render it by running `hugo` on the command line. Then send it to your Bonsai cluster with `curl`: [block:code] { "codes": [ { "code": "curl -H \"Content-Type: application/x-ndjson\" -XPOST \"https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_bulk\" --data-binary @public/bonsai.json", "language": "curl" } ] } [/block] You should now be able to see your data in the Elasticsearch cluster: [block:code] { "codes": [ { "code": "$ curl -XGET \"https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_search\"\n\n{\"took\":1,\"timed_out\":false,\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"hits\":{\"total\":1,\"max_score\":1.0,\"hits\":[{\"_index\":\"hugo\",\"_type\":\"doc\",\"_id\":...", "language": "shell" } ] } [/block]