Text Tokenization

Text Tokenization

Use this method to tokenize a text. This method handles non-word character spans like email addresses, hashtags, mentions, URLs, etc.

Method name is /Tokenize

Payload is a JSON object. A sample is given below.

{    
    "text": "Jon Snow, is the son of Lyanna Stark and Rhaegar Targaryen, the late Prince of Dragonstone."
}

Response is also a JSON object. A sample is given below.

{
  "tokens": [
    {
      "text": "Jon",
      "tag": "NNP"
    },
    {
      "text": "Snow",
      "tag": "NNP"
    },
    {
      "text": ",",
      "tag": ","
    },
    {
      "text": "is",
      "tag": "VBZ"
    },
    {
      "text": "the",
      "tag": "DT"
    },
    {
      "text": "son",
      "tag": "NN"
    },
    {
      "text": "of",
      "tag": "IN"
    },
    {
      "text": "Lyanna",
      "tag": "NNP"
    },
    {
      "text": "Stark",
      "tag": "NNP"
    },
    {
      "text": "and",
      "tag": "CC"
    },
    {
      "text": "Rhaegar",
      "tag": "NNP"
    },
    {
      "text": "Targaryen",
      "tag": "NNP"
    },
    {
      "text": ",",
      "tag": ","
    },
    {
      "text": "the",
      "tag": "DT"
    },
    {
      "text": "late",
      "tag": "JJ"
    },
    {
      "text": "Prince",
      "tag": "NN"
    },
    {
      "text": "of",
      "tag": "IN"
    },
    {
      "text": "Dragonstone",
      "tag": "NNP"
    },
    {
      "text": ".",
      "tag": "."
    }
  ]
}

View this method on RapidAPI.com

Text Summarization